noamaanMulla-03/transformers-issues huggingface dataset:
pipeline.from_pretrained
Author: cronoik
Pull request: no

# 🚀 Feature request

Nearly everyone who is using the transformers library is aware of the `from_pretrained()` and `save_pretrained()` concept. The [Pipeline class](https://huggingface.co/transformers/main_classes/pipelines.html#parent-class-pipeline) is currently only providing the `save_pretrained()` method which can cause confusion for some users as saving and loading of the pipeline needs to be done like this:
```
from transformers import pipeline

TASK = 'something'
DIRECTORY='something'
classifier = pipeline(TASK)
classifier.save_pretrained(DIRECTORY)  
c2 = pipeline(task = TASK, model=DIRECTORY, tokenizer=DIRECTORY)
``` 
This is probably not that obvious for people who just read the documentation and not the code. I suggest implementing the `from_pretrained()` method for the pipelines to make the library even more intuitive. 

## Your contribution

There is actually not much to do since the tokenizer and model are loaded by the corresponding Auto classes. The only information that is missing to construct the saved pipeline with `pipeline.from_pretraind(PATH)` is the task identifier. This information should be stored in a new separate file called `pipeline_config.json`.

I can provide a PR if you think this is a useful enhancement for the library @LysandreJik




Comments:
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.
Fix examples in M2M100 docstrings
Author: lewtun
Pull request: yes

# What does this PR do?

Replaces `tok` with `tokenizer` so examples can run with copy-paste




## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- albert, bert, xlm: @LysandreJik
- blenderbot, bart, marian, pegasus, encoderdecoder,  t5: @patrickvonplaten, @patil-suraj
- longformer, reformer, transfoxl, xlnet: @patrickvonplaten
- fsmt: @stas00
- funnel: @sgugger
- gpt2: @patrickvonplaten, @LysandreJik
- rag: @patrickvonplaten, @lhoestq
- tensorflow: @LysandreJik

Library:

- benchmarks: @patrickvonplaten
- deepspeed: @stas00
- ray/raytune: @richardliaw, @amogkam
- text generation: @patrickvonplaten
- tokenizers: @n1t0, @LysandreJik
- trainer: @sgugger
- pipelines: @LysandreJik

Documentation: @sgugger

HF projects:

- datasets: [different repo](https://github.com/huggingface/datasets)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Examples:

- maintained examples (not research project or legacy): @sgugger, @patil-suraj
- research_projects/bert-loses-patience: @JetRunner
- research_projects/distillation: @VictorSanh

 -->

@patil-suraj


Comments:
Implement Fast Tokenization for Deberta
Author: ShubhamSanghvi
Pull request: yes

# What does this PR do?

Fixes #10498 

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [x] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

@LysandreJik

Comments:
@LysandreJik , most of it was easy to figure out by looking at other tokenizers. The setup and testing guidelines were very easy to follow, I was up and running very quickly.

For the fast tokenizers, few things that might help someone like me who is new to the transformers library:
1. A top-level difference between the fast and slow tokenizers. At first, I did not know there was a tokenization library and took some time to figure that out.
2. Overview of how we implement tokenizers. Things like what do the vocab files do and what does the merges_file do. (Although this could just be me.)
3. In some fast tokenizers, there are files listed which are not being used. That confused me initially as I thought we needed those files for a fast tokenizer.

Hope this helps. 


@LysandreJik, 

#10498 mentioned implementing a tokenizer for deberta v2 as well. I have created a new feature request #11529 for that.

Thank you @ShubhamSanghvi, this is all very helpful. We'll take care of including that in the documentation so that it's clearer from now on.

Will take a look at #11529!
XLM-R tokenizer is none
Author: aggiejiang
Pull request: no

## Environment info

- `transformers` version: 4.3.2
- Platform: Linux-4.19.112+-x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.6.9
- PyTorch version (GPU?): 1.7.0+cu101 (True)
- Tensorflow version (GPU?): 2.4.1 (True)

### Who can help
@LysandreJik @n1t0

## Information

I am using XLM-R:

The problem arises when using:
* the official example scripts: (give details below)

The tasks I am working on is:
* my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behaviour:
```
tokenizer = XLMRobertaTokenizer.from_pretrained('xlm-roberta-base')
model = XLMRobertaModel.from_pretrained('xlm-roberta-base')
print(tokenizer, model)
```

## Result

The xlm-r tokenizer is none but the model can be found. 

I am a beginner for this model. Many thanks for your help.


Comments:
Hello! This is weird, it shouldn't happen. Could you try to install `sentencepiece` and let me know if it fixes your issue? Thank you!

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.

I'm still facing this issue.

![Screenshot from 2021-05-01 16-39-09](https://user-images.githubusercontent.com/12872080/116780709-f7dc6100-aa9b-11eb-93f5-e2ab59f67035.png)


Do you have a reproducible colab notebook? Thanks

Not sure how but it's working today.
![image](https://user-images.githubusercontent.com/12872080/116853234-5d069280-ac13-11eb-8cc9-4fa5098e27f8.png)


This is probably because you hadn't restarted your kernel after installing the `sentencepiece` dependency!
[Flax BERT/Roberta] few small fixes
Author: patil-suraj
Pull request: yes

# What does this PR do?

This PR fixes a few docs in Flax models.

Comments:
nit question: Any reason why we are accepting `bias_init` as a parameter while in all other modules we just use directly in `setup`
https://github.com/huggingface/transformers/blob/a5d2967bd8a5ed2456c593fa9eb5d9c0d726ae7a/src/transformers/models/bert/modeling_flax_bert.py#L480-L483
transformers.TFTrainer: Does not support batches with sequences of variable lengths?
Author: JulesGM
Pull request: no
Labels: name: wontfix

Hello,
It seems like `np.append` in `TFTrainer.prediction_loop` is the only thing that prevent TFTrainer from being able to deal with batches of variable sequence length (between the batches, not inside the batches themselves). Indeed, `np.append` requires the batches to be of the same sequence length. 
Alternatives: as this is in tensorflow, an easy alternative would be to convert the batches to `tf.RaggedTensor` with `tf.ragged.constant`, and to concatenate them (the usual way) with `tf.concat`.
You could also ofc just make `preds` and `label_ids` into lists. There doesn't seem to be any big computation going on on these objects.


Comments:
Created pull request https://github.com/huggingface/transformers/pull/8485

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


Hello @JulesGM 
Regarding [#8483](https://github.com/huggingface/transformers/pull/8485)
I went through the code and I found this helpful. But I'm facing issues to convert my training tf.data.Dataset to tf.RaggedTensor format. if possible can you share resources regarding this?
Adds Flax BERT finetuning example on GLUE
Author: marcvanzee
Pull request: yes

# What does this PR do?

Adds Flax BERT finetuning example which finetunes on one of the GLUE tasks.

I evaluated all tasks 5 times and added the average runs, the best run. and stdev in a table in the README. I used the seed of the best run as the default.

I also ran all experiments on three devices: 8 Cloud TPU-v3, 1 Cloud TPU-v3, 1 P100 GPU. I compared the runtimes and put them in another table in the README.

This PR was discussed over Slack with @patrickvonplaten and @sgugger .

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [X] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [X] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

Comments:
fix the mlm longformer example by changing [MASK] to <mask>
Author: fredo838
Pull request: yes

This PR fixes an official Huggingface example in the Longformer model

## Before submitting
- [x ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).

@Rocketknight1 this PR is to address https://github.com/huggingface/transformers/issues/11488#issuecomment-830065245

Is this what you meant? 



Comments:
Is there any way to look at the "new" docs locally? Because I don't know how to test my change if it's docs related

Yes! You can run `make docs` at the root of your clone. You can also run `make html` in the `docs` folder :)

You'll need to do `pip install .[docs]` beforehand to have all the appropriate dependencies.

thanks! It's looking good: 
![image](https://user-images.githubusercontent.com/11276933/116860790-61767f80-ac02-11eb-8a53-5533669951a8.png)


Great job, thanks for this! I made one minor change - I ran the `black` code formatter to make sure the code fits our style guidelines. You can do this yourself in future by running `pip install -U -e .[quality]` in the project root to install the code formatters, and then `make style` to reformat the code, but it's a very minor detail.

Once the tests check out I'll merge it. Thanks again for making my job as a maintainer easier!

No problem, this is what open source is about :D
Add LUKE
Author: NielsRogge
Pull request: yes

# What does this PR do?

It adds [LUKE: Deep Contextualized Entity Representations with Entity-aware Self-attention](https://arxiv.org/abs/2010.01057) by Yamada et al. [EMNLP 2020].

LUKE is similar to RoBERTa, but it adds an entity embedding matrix (500k Wikipedia entities!) and an entity-aware self-attention mechanism to improve performance on several downstream tasks that involve reasoning about entities, such as entity typing and relation classification. It was pre-trained using MLM on both tokens and entities from Wikipedia. 

Credits for this PR go to the original author @ikuyamada, who implemented everything. I've just set up everything he needed (a basic modeling file, configuration, conversion script, test files, etc.), and guided him through the process. 

Models are already on the hub: https://huggingface.co/models?search=studio-ousia

There are 3 head models defined: 
- `LukeForEntityClassification`, for tasks such as entity typing (given an entity in a sentence, classify it), e.g. the [Open Entity dataset](https://www.cs.utexas.edu/~eunsol/html_pages/open_entity.html). 
- `LukeForEntityPairClassification`, for tasks such as relation classification (classifying the relationship between two entities), e.g. the [TACRED dataset](https://nlp.stanford.edu/projects/tacred/).
- `LukeForEntitySpanClassification`, for tasks such as NER (LUKE obtains SOTA on NER! It considers all possible entity spans in a sentence, and then classifies them accordingly), e.g. the CoNLL-2003 dataset. 

To do:
- [x] add model cards (@ikuyamada this means adding READMEs to the models on the hub, you can take [BERT's one](https://huggingface.co/bert-base-uncased) as inspiration)
- [x] upload fine-tuned models to the hub

## Who can review?

@LysandreJik @sgugger 

Original Github conversation on the original repo: https://github.com/studio-ousia/luke/issues/38

Fixes #10700


Comments:
@LysandreJik @sgugger all comments are addressed, CI is green! Incredible work by the original author. 

Thanks, addressed your comments.

Added LUKE to the README, and added 3 community notebooks. 

Thanks again for all your work on this!
How to set up a custom tokenizer for distilbart
Author: neptune233
Pull request: no

Hi, I am currently using the distillbart. I pretrained a bart-large model with my own Chinese corpus and I just simply map each character to an id when I do the pre-training. In the distillition.py, the path or name of the tokenizer should be defined. My question is how can I build a custom tokenizer class which can be used in the code? Assume that we get a word dictionary which map character to the input id when I do the pre-training. 

Thank you.

Comments:
Hello, thanks for opening an issue! We try to keep the github issues for bugs/feature requests.
Could you ask your question on the [forum](https://discusss.huggingface.co) instead?

Thanks!
Update training tutorial
Author: sgugger
Pull request: yes

# What does this PR do?

This PRs rewrites the training tutorial that needed a bit of a refresher. It uses a simple example on the IMDB dataset (for basic text classification) with fine-tuning using:
- Trainer
- Keras
- Raw training loop in PyTorch

One last section with the raw training loop in TensorFlow could be added in a follow-up PR.

Comments:
Re: freezing layers, someone pointed me to your answer here as well: From your deepspeed talk: https://youtu.be/RG-yV5zgqjQ?t=2450

I guess it's still a bit counterintuitive that the randomly initialized head doesn't cause havoc on the weights of base layers?  Just to make sure, it's not that you unfreeze the whole thing after only training the head first?  You just train the entire thing unfrozen from the very beginning?  

I've shared this with several folks and there was enough surprise that it could be worth mentioning this explicitly in this tutorial, as this is something I think lots of folks may not know!  

P.S. if it is the case that you unfreeze the entire thing from the very beginning this is such a surprising result I feel like it's worth a blog post or something!  

This is a surprise for someone who comes from the fastai community but insisting a lot on this for a user who don't even know what freezing layers is won't be helpful either, which is why I'm not mentioning it in the new version. I'll try to find a compromise between the two :-)

And yes, Transformers model are usually fine-tuned without freezing anything, which is what is done in all the research papers for GLUE/Squad etc. Training only the randomly initialized head usually does not achieve anything good and the state it ends in is so bad you can't recover by fine-tuning the whole model after.

@sgugger thank you for clarifying, and thanks for your patience!  I'm really glad I learned about this today as I have been doing it wrong all along.  

Seems like an interesting research project to find out "why"
[Deepspeed] fix resize_token_embeddings
Author: stas00
Pull request: yes

the introduction of `model.resize_token_embeddings(len(tokenizer))` in https://github.com/huggingface/transformers/commit/57c8e822f7faa1c19f9926338f21f3aab2269997#diff-09777f56cee1060a535a72ce99a6c96cdb7f330c8cc3f9dcca442b3f7768237a
uncovered a bug in Deepspeed integration. 

We have to create the new resized embedding with the same dtype as the original.
If not deepspeed will fail to initialize with: `"Model must initialized in fp16 mode for ZeRO Stage 3.` error.

This PR fixes this bug.

@sgugger 
 

Comments:
longform
Author: AprilSongRits
Pull request: no

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version:
- Platform:
- Python version:
- PyTorch version (GPU?):
- Tensorflow version (GPU?):
- Using GPU in script?:
- Using distributed or parallel set-up in script?:

### Who can help
<!-- Your issue will be replied to more quickly if you can figure out the right person to tag with @
 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- albert, bert, xlm: @LysandreJik
- blenderbot, bart, marian, pegasus, encoderdecoder,  t5: @patrickvonplaten, @patil-suraj
- longformer, reformer, transfoxl, xlnet: @patrickvonplaten
- fsmt: @stas00
- funnel: @sgugger
- gpt2: @patrickvonplaten, @LysandreJik
- rag: @patrickvonplaten, @lhoestq
- tensorflow: @Rocketknight1

Library:

- benchmarks: @patrickvonplaten
- deepspeed: @stas00
- ray/raytune: @richardliaw, @amogkam
- text generation: @patrickvonplaten
- tokenizers: @LysandreJik
- trainer: @sgugger
- pipelines: @LysandreJik

Documentation: @sgugger

Model hub:

- for issues with a model report at https://discuss.huggingface.co/ and tag the model's creator.

HF projects:

- datasets: [different repo](https://github.com/huggingface/datasets)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Examples:

- maintained examples (not research project or legacy): @sgugger, @patil-suraj
- research_projects/bert-loses-patience: @JetRunner
- research_projects/distillation: @VictorSanh

 -->

## Information

Model I am using (Bert, XLNet ...):

The problem arises when using:
* [ ] the official example scripts: (give details below)
* [ ] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [ ] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:

1.
2.
3.

<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

## Expected behavior

<!-- A clear and concise description of what you would expect to happen. -->


Comments:
I want to create a tokenizer which the vocab file in my computer
Author: xiongma
Pull request: no

I want to create a tokenizer in my computer, which the vocab file is below, my code as below
```
self.model_config = AutoConfig.from_pretrained(r'/home/cally/Awake/Code/bert/pre_model/bert_config.json')
self.tokenizer = AutoTokenizer.from_pretrained(r'/home/cally/Awake/Code/bert/pre_model/vocab.txt')
```
it appeared this error, what's wrong?, my enviroment is linux python 3.7 transformer 2.6
```
OSError: Couldn't reach server at '/home/cally/Awake/Code/bert/pre_model/vocab.txt' to download configuration file or configuration file is not a valid JSON file. Please check network or file content here: /home/cally/Awake/Code/bert/pre_model/vocab.txt.

```

Comments:
Your config.json file should be named `config.json`, and then you should be able to do:
```
self.model_config = AutoConfig.from_pretrained(r'/home/cally/Awake/Code/bert/pre_model/')
self.tokenizer = AutoTokenizer.from_pretrained(r'/home/cally/Awake/Code/bert/pre_model/')
```

(just the folder name).

You might have to add `model_type: "bert"` to your config.json.

> 
> 
> Your config.json file should be named `config.json`, and then you should be able to do:
> 
> ```
> self.model_config = AutoConfig.from_pretrained(r'/home/cally/Awake/Code/bert/pre_model/')
> self.tokenizer = AutoTokenizer.from_pretrained(r'/home/cally/Awake/Code/bert/pre_model/')
> ```
> 
> (just the folder name).
> 
> You might have to add `model_type: "bert"` to your config.json.

I encountered the same problem and sorry to bother, but where should i add this line to?
Enable added tokens
Author: LysandreJik
Pull request: yes

Currently, the only way to manage adding `AddedToken`s to a tokenizer is via the `tokenizer.add_special_tokens` or `tokenizer.add_tokens` methods; it should also be enabled from the initialization.

Previously this was impossible:

```py
special_tokens = [AddedToken('<special>')]

GPT2Tokenizer.from_pretrained('gpt2', additional_special_tokens=special_tokens)
```

This enables this functionality and adds a test. This also fixes all tokenizers that were ill-configured for that purpose..

Comments:
@sgugger, please merge if you're happy with the updated PR :)