noamaanMulla-03/transformers-issues huggingface dataset:
Is there a way of finetuning DistilGPT2?
Author: aditya1702
Pull request: no
Labels: name: wontfix

## ❓ Questions & Help

<!-- A clear and concise description of the question. -->
Hello! I am trying to finetune GPT2 for a project of mine but my aim is to deploy the model on a server and hence, I would like the final model file to be small. I was thinking of using the Distil* models especially the DistilGPT2 but couldnt find it in the run_lm_finetuning.py script. 

Currently, is finetuning distilGPT2 supported? If yes, then how should I do it?

Comments:
Yes you can, call `run_lm_finetuning` via these parameters:

```
--model_type gpt2 \
--model_name_or_path distilgpt2 
```

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


I can't find the file `run_lm_finetuning.py` under this repo. Could you please give the full file path please? Thanks a lot.

> I can't find the file `run_lm_finetuning.py` under this repo. Could you please give the full file path please? Thanks a lot.

Got it . it has been renamed to [run_language_modeling.py](https://github.com/huggingface/transformers/blob/master/examples/language-modeling/run_language_modeling.py) now.
Dataset Lazyloader for transformers trainer
Author: agemagician
Pull request: no

# 🚀 Feature request

Currently, train/fine-tune models using transformers trainer loads the whole dataset in memory.
This could work for small datasets, but with large datasets the training will crash.

Code:
https://github.com/huggingface/transformers/blob/master/examples/language-modeling/run_language_modeling.py#L128

## Motivation

We have trained 0.4 Billion Bert Model and currently training a 1.8 Billion Bert Model for protein sequences, and we want to train DistilBERT to reduce the model size.

Unfortunately, our dataset is very huge about 0.7 Terabyte and since the trainer loads the whole dataset the trainer crashes.

It will be more optimised if you could use lazy loader for loading the datasets like fairseq.

@patrickvonplaten  I am not sure who is responsible for the data loader for LM.

Comments:
As far as I can see there is currently no dataloader functionality to lazy load data into memory. 
It should not be very hard to implement such a feature yourself, I think, see https://discuss.pytorch.org/t/loading-huge-data-functionality/346/2 .
Also cc @lhoestq @sgugger @thomwolf - Are we planning on providing more support for lazy data loading? 

There is an [open PR #4009](https://github.com/huggingface/transformers/pull/4009) about this that was mostly close to being merged (cc @BramVanroy) and I must confess I kind of dropped the ball on it.

Shouldn't be a ton of work to complete it and get it merged.

As mentioned in that PR, the other option is to use `huggingface/nlp` which can load large text datasets lazily out of the box.

@patrickvonplaten @julien-c Thanks a lot for your reply.

Should we close this issue and focus on https://github.com/huggingface/transformers/pull/4009  ?

The long-term solution is to use nlp for this IMO.

I have checked NLP and it is slow, maybe I am doing something wrong.
I made a simple python script to check it is speed, which loads 1.1 TB of textual data.
It has been 8 hours and still, it is on the loading steps.
It does work when the text dataset size is small about  1 GB, but it doesn't scale.
It also uses a single thread during the data loading step.

```
train_files = glob.glob("xxx/*.txt",recursive=True)
random.shuffle(train_files)

print(train_files)

dataset = nlp.load_dataset('text', 
                           data_files=train_files,
                           name="customDataset",
                           version="1.0.0",
                           cache_dir="xxx/nlp")
```

You should open this issue on the nlp repo, to make sure it's not forgotten! In particular they are working a lot on performance right now :-)

done:
https://github.com/huggingface/nlp/issues/546


I am closing this in favour of:
- https://github.com/huggingface/transformers/pull/4009
- https://github.com/huggingface/nlp/issues/546
head_mask in modeling_bert.py
Author: franciszzj
Pull request: no

Should change https://github.com/huggingface/transformers/blob/master/src/transformers/modeling_bert.py#L487 `head_mask[i]` to `head_mask[i] if head_mask is not None else None`.

full code:

```
class BertEncoder(nn.Module):
    def __init__(self, config):
        super().__init__()
        self.config = config
        self.layer = nn.ModuleList([BertLayer(config) for _ in range(config.num_hidden_layers)])

    def forward(
        self,
        hidden_states,
        attention_mask=None,
        head_mask=None,
        encoder_hidden_states=None,
        encoder_attention_mask=None,
        output_attentions=False,
        output_hidden_states=False,
        return_dict=False,
    ):
        all_hidden_states = () if output_hidden_states else None
        all_attentions = () if output_attentions else None
        for i, layer_module in enumerate(self.layer):
            if output_hidden_states:
                all_hidden_states = all_hidden_states + (hidden_states,)

            if getattr(self.config, "gradient_checkpointing", False):

                def create_custom_forward(module):
                    def custom_forward(*inputs):
                        return module(*inputs, output_attentions)

                    return custom_forward

                layer_outputs = torch.utils.checkpoint.checkpoint(
                    create_custom_forward(layer_module),
                    hidden_states,
                    attention_mask,
                    head_mask[i] if head_mask is not None else None,
                    encoder_hidden_states,
                    encoder_attention_mask,
                )
            else:
                layer_outputs = layer_module(
                    hidden_states,
                    attention_mask,
                    head_mask[i] if head_mask is not None else None,
                    encoder_hidden_states,
                    encoder_attention_mask,
                    output_attentions,
                )
            hidden_states = layer_outputs[0]
            if output_attentions:
                all_attentions = all_attentions + (layer_outputs[1],)

        if output_hidden_states:
            all_hidden_states = all_hidden_states + (hidden_states,)

        if not return_dict:
            return tuple(v for v in [hidden_states, all_hidden_states, all_attentions] if v is not None)
        return BaseModelOutput(
            last_hidden_state=hidden_states, hidden_states=all_hidden_states, attentions=all_attentions
        )
```

Comments:
Runtime for BERT and Roberta
Author: AkshitaJha
Pull request: no
Labels: name: wontfix

I'd like to train a BERT model from scratch. Approximately, how long should it take to train 800k sentences (batch size of say 32) on a 10GB GeForce RTX 2080 GPU? 

If I just fine-tune BERT for 800k sentences for 4 epochs, how long should that take?

Are there any benchmarks available except [exacct](https://blog.exxactcorp.com/nvidia-quadro-rtx-6000-bert-large-fine-tune-benchmarks-with-squad-dataset/)? 

How much faster is RoBerta?

Comments:
Which version of bert or Roberta you want to use base, large ? It also depends on maximum sequence length

I'd like to use the base version with the maximum sequence length of 128.

Hi @AkshitaJha , you can run training for few steps 1 or 2, that should give you rough idea of how much time it'll take to finish one epoch

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
t5 embed_tokens
Author: agnesmm
Pull request: no

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->
     
- `transformers` version:
- Platform:
- Python version:
- PyTorch version (GPU?):
- Tensorflow version (GPU?):
- Using GPU in script?:
- Using distributed or parallel set-up in script?:

### Who can help
<!-- Your issue will be replied to more quickly if you can figure out the right person to tag with @
 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.
 
 albert, bert, GPT2, XLM: @LysandreJik 
 tokenizers: @mfuntowicz
 Trainer: @sgugger
 Speed and Memory Benchmarks: @patrickvonplaten
 Model Cards: @julien-c
 Translation: @sshleifer
 Summarization: @sshleifer
 TextGeneration: @TevenLeScao 
 examples/distillation: @VictorSanh
 nlp datasets: [different repo](https://github.com/huggingface/nlp)
 rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)
 Text Generation: @TevenLeScao
 blenderbot: @mariamabarham
 Bart: @sshleifer
 Marian: @sshleifer
 T5: @patrickvonplaten
 Longformer/Reformer: @patrickvonplaten
 TransfoXL/XLNet: @TevenLeScao 
 examples/seq2seq: @sshleifer
 examples/bert-loses-patience: @JetRunner
 tensorflow: @jplu
 examples/token-classification: @stefan-it
 documentation: @sgugger
 -->

## Information

Model I am using (Bert, XLNet ...):

The problem arises when using:
* [ ] the official example scripts: (give details below)
* [ ] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [ ] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:

1.
2.
3.

<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

## Expected behavior

<!-- A clear and concise description of what you would expect to happen. -->


Comments:
onnx-export example notebook is failing for TF
Author: Zhen-hao
Pull request: no

hi, I'm using the latest 3.1.0 release.

When I run
```
from transformers.convert_graph_to_onnx import convert

# Tensorflow 
convert(framework="tf", model="bert-base-cased", output="onnx/bert-base-cased.onnx", opset=11)
```
as shown in https://github.com/huggingface/transformers/blob/master/notebooks/04-onnx-export.ipynb,

the following error occurs
```
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-1-bc5982e91176> in <module>
      7 
      8 # Tensorflow
----> 9 convert(framework="tf", model="bert-base-cased", output="onnx/bert-base-cased.onnx", opset=11)

/nix/store/w8xw8jng4dfjcqijfjw1sps8pim669kj-python3.7-transformers-3.1.0/lib/python3.7/site-packages/transformers/convert_graph_to_onnx.py in convert(framework, model, output, opset, tokenizer, use_external_format, pipeline_name)
    335     nlp = load_graph_from_args(pipeline_name, framework, model, tokenizer)
    336 
--> 337     if not output.parent.exists():
    338         print(f"Creating folder {output.parent}")
    339         makedirs(output.parent.as_posix())

AttributeError: 'str' object has no attribute 'parent'
```

Comments:
The following should work:

```
from pathlib import Path
from transformers.convert_graph_to_onnx import convert

# Tensorflow 
convert(framework="tf", model="bert-base-cased", output=Path("onnx/bert-base-cased.onnx"), opset=11)
```

@subho406  thanks! I thought the error was about the model output since using string in output path had worked in the previous version.
Create README.md
Author: abedkhooli
Pull request: yes
Labels: name: model card

<!-- This line specifies which issue to close after the pull request is merged. -->
Model card for akhooli/mbart-large-cc25-en-ar

Comments:
@sshleifer want to update the inference API so that the correct pipeline shows up at https://huggingface.co/akhooli/mbart-large-cc25-en-ar ? (cc @mfuntowicz)

Seems fixed?
https://huggingface.co/akhooli/mbart-large-cc25-en-ar
![image](https://user-images.githubusercontent.com/6045025/92133236-22273300-edd6-11ea-866e-d7249f38f792.png)


> 
> 
> Seems fixed?
> https://huggingface.co/akhooli/mbart-large-cc25-en-ar
> ![image](https://user-images.githubusercontent.com/6045025/92133236-22273300-edd6-11ea-866e-d7249f38f792.png)
Sure, just after the model card was merged. Not sure if it was due to the 'translation' tag in the card or some other magic done by your team.

Just uploaded https://huggingface.co/akhooli/mbart-large-cc25-ar-en and it seems inference type is not recognized automatically. It defaults to fill-mask (model card submitted).

model card merged.
TypeError: __init__() got an unexpected keyword argument 'cache_dir'
Author: Abbyyan
Pull request: no


I'm fine tuning with the example `run_language_modeling.py`as follows.
```shell
python run_language_modeling.py --output_dir=output_dir --model_type gpt2 --model_name_or_path distilgpt2  --do_train  --train_data_file=xxx.data.txt
```
It failed with error 
```shell
Traceback (most recent call last):
  File "run_language_modeling.py", line 313, in <module>
    main()
  File "run_language_modeling.py", line 242, in main
    get_dataset(data_args, tokenizer=tokenizer, cache_dir=model_args.cache_dir) if training_args.do_train else None
  File "run_language_modeling.py", line 143, in get_dataset
    cache_dir=cache_dir,
TypeError: __init__() got an unexpected keyword argument 'cache_dir'
```
Could you please tell me how to use it ? Thanks a lot.
<!-- A clear and concise description of what you would expect to happen. -->


Comments:
Solved with  https://github.com/huggingface/transformers/issues/319
Mobile Bert Tiny model
Author: borsork377
Pull request: no

# 🚀 Feature request

Can you add support for variants of MobileBERT?

## Motivation

The package currently provides various varients of bert - 'bert-base-cased', 'bert-base-uncased', 'bert-large'...
Can you also similarly provide Mobile Bert Tiny as well?



Comments:
MobileBERT is supported, see [here](https://github.com/huggingface/transformers/blob/master/src/transformers/modeling_mobilebert.py) for the code, and [here](https://huggingface.co/transformers/model_doc/mobilebert.html) for the docs. [Here](https://huggingface.co/models?search=mobilebert) are all the available mobilebert models on the hub.
How to build Bimodel to search code snippets? [CodeBERTa]
Author: hmdgit
Pull request: no
Labels: name: Discussion

name: wontfix

Hi,

I would like build a code search engine model. The main purpose is that when I pass docstring, it should give me top-k associated code snippets as results. 

I have a data in the form of (docstring, code), which means each docstring is associated with mentioned code snippet.

I have seen CodeBERTa fine tune [code](https://huggingface.co/huggingface/CodeBERTa-language-id), but it is not using docstring in it. Is it possible to use this model?

Can you please give me some entry points to solve this problem by using hugging-face library?




Comments:
CodeBERTa was indeed trained on just the code so you would need to tweak the approach.

Did you read the paper for CodeSearchNet (https://arxiv.org/abs/1909.09436) by @hamelsmu?

Thanks Julien for your response.

I have taken an overview of the paper and its [code](https://github.com/github/CodeSearchNet), and I will try it.

But, can it be possible to solve it by using BERT huggingface library? What kind of tweaks do I need to apply in CodeBERT  fine tune [code](https://huggingface.co/huggingface/CodeBERTa-language-id)? 

Can it be solved by finetuning BertForQuestionAnswering [code](https://huggingface.co/transformers/model_doc/bert.html#bertforquestionanswering)?

Maybe CodeBERT([https://arxiv.org/abs/2002.08155](https://arxiv.org/abs/2002.08155)) is suitable for you.

This paper is of my high interest....
Is there fine tuning source code for that paper publicly available? or are there any short snippets available which can help in fine-tuning?


You can visit this link ([https://github.com/microsoft/CodeBERT](https://github.com/microsoft/CodeBERT)) . 

Thanks for sharing. I will check and let you know about related concerns on the shared github repository

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
typo
Author: stas00
Pull request: yes

there is no var `decoder_input_ids`, but there is `input_ids` for decoder :)


Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/6959?src=pr&el=h1) Report
> Merging [#6959](https://codecov.io/gh/huggingface/transformers/pull/6959?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/56742e9f610231d7b28fe2387770dc56014b79de?el=desc) will **decrease** coverage by `0.01%`.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/6959/graphs/tree.svg?width=650&height=150&src=pr&token=9qOlN6Hb1c)](https://codecov.io/gh/huggingface/transformers/pull/6959?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master    #6959      +/-   ##
==========================================
- Coverage   80.00%   79.98%   -0.02%     
==========================================
  Files         161      161              
  Lines       30120    30120              
==========================================
- Hits        24097    24092       -5     
- Misses       6023     6028       +5     
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/6959?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [src/transformers/generation\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/6959/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9nZW5lcmF0aW9uX3V0aWxzLnB5) | `97.21% <ø> (ø)` | |
| [src/transformers/generation\_tf\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/6959/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9nZW5lcmF0aW9uX3RmX3V0aWxzLnB5) | `83.45% <0.00%> (-1.76%)` | :arrow_down: |
| [src/transformers/file\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/6959/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9maWxlX3V0aWxzLnB5) | `82.66% <0.00%> (+0.25%)` | :arrow_up: |
| [src/transformers/data/data\_collator.py](https://codecov.io/gh/huggingface/transformers/pull/6959/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9kYXRhL2RhdGFfY29sbGF0b3IucHk=) | `92.30% <0.00%> (+0.40%)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/6959?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/6959?src=pr&el=footer). Last update [56742e9...12a1792](https://codecov.io/gh/huggingface/transformers/pull/6959?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
The output to be used for getting sentence embeddings from BERT
Author: AkshitaJha
Pull request: no
Labels: name: wontfix

What is the output that we should be using to get embeddings for a sentence using BERT? When I load the pre-trained BERT model ([BertModel](https://huggingface.co/transformers/model_doc/bert.html#transformers.BertModel)) from huggingface for inference, should I be using the `pooler_output', the output of the last hidden layer or something else?

While fine-tuning BERT, which huggingface module should be used for getting sentence embeddings? Is it the [BertForSequenceClassification](https://huggingface.co/transformers/model_doc/bert.html#bertforsequenceclassification),  [BertForMaskedLM](https://huggingface.co/transformers/model_doc/bert.html#bertformaskedlm), [BertModel](https://huggingface.co/transformers/model_doc/bert.html#bertmodel), or some other module? 

Comments:
Hi @AkshitaJha , what is your downstream task ? 
Also you may wanna try this out for sentence embeddings 
https://huggingface.co/deepset/sentence_bert
https://github.com/UKPLab/sentence-transformers


This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
README for HooshvareLab/bert-fa-base-uncased
Author: m3hrdadfi
Pull request: yes
Labels: name: model card

ParsBERT v2.0 is a fine-tuned and vocab-reconstructed version of ParsBERT, and it's able to be used in other scopes!

Some features:
- We added some unused-vocab for use in summarization and other scopes.
- We fine-tuned the model on vast styles of writing in the Persian language.


Comments:
🐛 T5 Tokenizer ignores \n \t characters and more than one whitespace together
Author: misrasaurabh1
Pull request: no

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->
     
- `transformers` version: 3.0.2 (master)
- Platform: Linux-4.9.0-12-amd64-x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.6.9
- PyTorch version (GPU?): 1.5.1+cpu (False)
- Tensorflow version (GPU?): not installed (NA)
- Using GPU in script?: no
- Using distributed or parallel set-up in script?: no

### Who can help

@patrickvonplaten @sshleifer

## Information
T5 Tokenizer is based out of SentencePiece and in sentencepiece Whitespace is treated as a basic symbol. But huggingface tokenizers just ignores more than one whitespace. 
Consider all the following examples tokenize to the same thing.
```
from transformers import T5Tokenizer
tokenizer = T5Tokenizer.from_pretrained("t5-base")

print(tokenizer.tokenize("Hi there I'm good"))
>> ['▁Hi', '▁there', '▁I', "'", 'm', '▁good']

print(tokenizer.tokenize("Hi there     I'm good"))
>> ['▁Hi', '▁there', '▁I', "'", 'm', '▁good']

print(tokenizer.tokenize("Hi there    I'm good\n"))
>> ['▁Hi', '▁there', '▁I', "'", 'm', '▁good']

print(tokenizer.tokenize("Hi there \n   I'm good\n"))
>> ['▁Hi', '▁there', '▁I', "'", 'm', '▁good']

print(tokenizer.tokenize("Hi there \n     I'm good\n"))
>> ['▁Hi', '▁there', '▁I', "'", 'm', '▁good']

print(tokenizer.tokenize("Hi there \n  \t   I'm good\n"))
>> ['▁Hi', '▁there', '▁I', "'", 'm', '▁good']

print(tokenizer.tokenize("Hi there\nI'm good"))
>> ['▁Hi', '▁there', '▁I', "'", 'm', '▁good']
```
All these examples should tokenize to different representations. Also ignoring newline outright means that all applications that use newlines fail.
Model I am using (Bert, XLNet ...): T5


<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

## Expected behavior

<!-- A clear and concise description of what you would expect to happen. -->
All the whitespaces have different tokenizations

Comments:
@TevenLeScao @sshleifer @patrickvonplaten This looks like a serious problem with the T5 Tokenizer. Is this behavior expected?

Closing this issue as sentencepiece for T5 removes more than one whitespace as a standard https://github.com/google-research/text-to-text-transfer-transformer/issues/390#issuecomment-688417703