noamaanMulla-03/transformers-issues huggingface dataset:
Running `run_ner_no_trainer.py` with `--label_all_tokens` falsifies seqeval results
Author: sven-nm
Pull request: no

## Environment info

- `transformers` version: 4.11.3
- Platform: Linux-5.4.0-80-generic-x86_64-with-glibc2.17
- Python version: 3.8.11
- PyTorch version (GPU?): 1.9.0 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: True
- Using distributed or parallel set-up in script?: False

## Who can help

Examples:

- maintained examples (not research project or legacy): @sgugger, @patil-suraj

## Description

Hi 🤗 team, not really an bug report, as the code is doing what it should, but not strictly a feature request either. 

When running `run_ner_no_trainer.py` with `--label_all_tokens`, evaluating with `seqeval` (as default in the script) completely fudges the results. This is due to the fact that `seqeval` is entity based, which means that each sub-token in a "B-" labelled word  will be considered as a single entity. 

Example : 

Before tokenisation: 
```
O    B-PERS      I-PERS  I-PERS
The  Australian  Prime   minister
```
-> Number of entities: 1


After tokenisation with `label_all_tokens`:
 ```
O    B-PERS   B-PERS   I-PERS  I-PERS
The  Austral  ##ian    Prime   minister
```
-> Number of entities: 2

This can yield pretty important differences which may easily be overseen. I think it is worth adding a warning when using the script with this configuration.

Comments:
Ah yes, we should convert the B- to an I-, which we do in other examples.
Fix assert in src/transformers/data/datasets/language_modeling.py
Author: skpig
Pull request: yes

# What does this PR do?

Replace assertion in src/transformers/data/datasets/language_modeling.py with ValueError
<!-- Remove if not applicable -->

Related to (#12789)


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?
@sgugger

Comments:
replace assert with exception in src/transformers/utils/model_pararallel_utils.py
Author: skpig
Pull request: yes

# What does this PR do?

<!--
-->
The PR replaces the assertions in src/transformers/utils/model_pararallel_utils.py with ValueError exceptions . I remain the error messages in the original code. @sgugger 

<!-- Remove if not applicable -->

Fixes # (#12789)


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


Comments:
Thanks a lot!
Add BEiT
Author: NielsRogge
Pull request: yes

# What does this PR do?

It adds [BEiT: BERT Pre-Training of Image Transformers](https://arxiv.org/abs/2106.08254) to the library. It's the first paper that enables self-supervised pre-trained Vision Transformers (ViTs) to outperform their supervised pre-training counterparts. As a picture says more than a thousand (or 16x16?) words, this is a good summary of the approach:

<img width="544" alt="Schermafbeelding 2021-08-03 om 17 26 19" src="https://user-images.githubusercontent.com/48327001/128042500-538a6fef-8d92-42b0-92f5-31d06ca6ae36.png">

The authors used OpenAI's [DALL-E](https://github.com/openai/DALL-E)'s encoder to map images to tokens, which the model then needs to predict based on masked patches. There are 3 models defined: `BEiTModel`, `BEiTForMaskedImageModeling` and `BEiTForImageClassification`.

This PR also cleans up some scripts from the library, namely those that defined id2label dicts for several datasets. I have removed `imagenet_classes.py` and `coco_classes.py` from the utils directory. Instead, id2label's are now defined on the hub in their [own repository](https://huggingface.co/datasets/huggingface/label-files). These can then be used in conversion scripts using the `huggingface_hub` library. 

## To do

- [x] Add all checkpoints to the hub, under the "Microsoft" namespace. Perhaps discuss the model names, because for example `microsoft/beit_base_patch16_224_pt22k_ft22k_to_1k` is getting out of hand
- [ ] Would be cool to have a working colab for the `BEiTForMaskedImageModeling` model. For this, tagging one of the original authors: @donglixp 

In a future PR, I also plan to add the semantic segmentation model, which obtains SOTA on Ade20k.


Comments:
I've uploaded all checkpoints to the hub: https://huggingface.co/models?search=microsoft/beit

I've renamed the checkpoints which are fine-tuned on ImageNet-1k (after being intermediately fine-tuned on ImageNet-22k) to be just `microsoft/beit-base-patch16-224`, etc.

@donglixp if you're interested, could you write model cards for these models? Model cards are READMEs that describe the models in detail. You can take inspiration from ViT's [model card](https://huggingface.co/google/vit-base-patch16-224). 

Also, I do have a notebook for `BEiTForMaskedImageModeling`, but it's not working as expected. Could you please take a look? https://colab.research.google.com/drive/1Mjt-3jHw9HYMXECmSdDlbiG59ZAw-Z0T?usp=sharing

@NielsRogge great work, any news on the future PR, to add the semantic segmentation model and the pretrained Ade20k? Thanks!

@JStumpp say no more, it's added ;)
saving the model after each epoch completion
Author: Aniruddha-JU
Pull request: no

 I am running run_summarization.py file with mt5 model, and wanted to save the check point after each epoch. So, I changed the trainer file in the bigbird_flax.py file, accordingly but it is not happening. I have attached file
![Capture](https://user-images.githubusercontent.com/36475622/138246764-638bed57-2afe-4dc3-899e-8ec9259487b5.PNG)


Comments:
Hi,

Can you ask this question on our [forum](https://discuss.huggingface.co/) please? We like to keep Github issues for bugs/feature requests.

Thanks!
Replace assertions with ValueError exceptions
Author: ddrm86
Pull request: yes

# What does this PR do?

Replaces the assertions in generation_utils.py with ValueError exceptions. I added an error message for the first one that did not exist before, feel free to change it.

Contributes towards fixing issue #12789


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


Comments:
TF CLM example fix typo
Author: Rocketknight1
Pull request: yes

Fixes a one-line typo in the TF CLM example - it was still using `MODEL_FOR_MASKED_LM_MAPPING`

Comments:
Also oop, this looks like it was based on a slightly older branch - it won't cause any problems, but the "Files changed" tab lists some changes in other files that are already merged - the only one actually affected is `run_clm.py`

There might be more such remnants from the MLM script, see https://github.com/huggingface/transformers/pull/14014

cc @Rocketknight1
RAG issue
Author: kingafy
Pull request: no

## Environment info. 
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version:4.8.2
- Platform:Windows 
- Python version:3.7.10
- PyTorch version (GPU?):CPU version 1.9.0
- Tensorflow version (GPU?):
- Using GPU in script?:no
- Using distributed or parallel set-up in script?:

Models:

- RAG
facebook/rag-sequence-nq

I am trying to use the retriever process of RAG mentioned in https://huggingface.co/transformers/model_doc/rag.html

I have created my own knowledge embedding using the process provided in the documentation. The process of retrieval works perfectly in transformer version transformers =4.6.1 but fails in the version 4.8.2

Issue is in the step:-tokenizer = RagTokenizer.from_pretrained(r"C:\My_Projects\code-repo\retriever\models")  ##local dir hosting the model facebook/rag-sequence-nq

ValueError: unable to parse C:\My_Projects\code-repo\retriever\models\tokenizer_config.json as a URL or as a local path.

Any help on the same would be appreciated as I have a feeling its a version issue. Do we need to implement it differently in the latest version.



Comments:
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.
Change asserts in src/transformers/models/xlnet/ to raise ValueError
Author: westonkl
Pull request: yes

# What does this PR do?


Replaces asserts in:
src/transformers/models/xlnet/configuration_xlnet.py and src/transformers/models/xlnet/modeling_tf_xlnet.py  
to a ValueError exception.

Contributes towards fixing issue #12789


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

### Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.
@sgugger 


Comments:
Fix ignore_mismatched_sizes
Author: qqaatw
Pull request: yes

# What does this PR do?


<!-- Remove if not applicable -->

Fixes #14073 




## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

@sgugger 

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- albert, bert, xlm: @LysandreJik
- blenderbot, bart, marian, pegasus, encoderdecoder,  t5: @patrickvonplaten, @patil-suraj
- longformer, reformer, transfoxl, xlnet: @patrickvonplaten
- fsmt: @stas00
- funnel: @sgugger
- gpt2: @patrickvonplaten, @LysandreJik
- rag: @patrickvonplaten, @lhoestq
- tensorflow: @LysandreJik

Library:

- benchmarks: @patrickvonplaten
- deepspeed: @stas00
- ray/raytune: @richardliaw, @amogkam
- text generation: @patrickvonplaten
- tokenizers: @n1t0, @LysandreJik
- trainer: @sgugger
- pipelines: @LysandreJik

Documentation: @sgugger

HF projects:

- datasets: [different repo](https://github.com/huggingface/datasets)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Examples:

- maintained examples (not research project or legacy): @sgugger, @patil-suraj
- research_projects/bert-loses-patience: @JetRunner
- research_projects/distillation: @VictorSanh

 -->


Comments:
As you can see from the multiple failing tests, this makes the use case where we have a model with a task-specific head fail, so it looks like the fix is more complicated than just swapping the lines.

I believe these failures are due to added AutoModel (TFAutoModel / FlaxAutoModel) test cases. I'm finding an appropriate output of AutoModel that can test mismatched sizes, but it seems that there are some model-specific restrictions or assertions.

Indeed, some of the models are failing because of inner math between the hidden size and others, whereas others fail differently. You should adapt your test to just the vocab_size maybe?

Thanks for fixing the tests! You should also ignore the test for LayoutLmv2 (who wants a `bbox`) and there is the encoder-decoder template test to fix, I left a suggestion for that.

Fixed, thanks for the suggestions.

I agree with @patrickvonplaten. Maybe we can rename `add_prefix` to `add_prefix_to_init_model` as well to make it more clear.

I agree with @patrickvonplaten suggestion. Let's merge this PR once the comment is addressed, and then we can do the renaming in a followup PR!

You will need to include [this commit](https://github.com/huggingface/transformers/commit/0f502682fb08c7dac5b255903f63dcd9cc2d68eb) in your PR branch as the latest release of PyTorch 1.10 broke our CI.
Auto model for conditional generation
Author: sadakmed
Pull request: no

# 🚀 Feature request

The majority of tasks (CausalLM, QuestionAnswering...) have the option to be loaded without importing the specific architecture of the model with `AutoModel` yet for  `ConditionalGeneration`  you need to import the specific class.

is there a technical reason restraining such an implementation?


Comments:
If you can confirm that this would be a good plus, I can push my modification for this.

Hello! These models are available in `AutoModelForSeq2SeqLM`!

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.
Replace assertion with ValueError exception
Author: ddrm86
Pull request: yes

Replaces the assertion in hf_argparser.py with a ValueError exception.

Contributes towards fixing issue #12789

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


Comments:
Replace assert statements with exceptions (#13871)
Author: ddrm86
Pull request: yes

# What does this PR do?

Replaces the assertions in generation_beam_search.py with ValueError exceptions. I added an error message for the first one that did not exist before, feel free to change it.

Contributes towards fixing issue #12789

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


Comments:
Translate README.md to Korean
Author: yeounyi
Pull request: yes

# What does this PR do?
Add README_ko.md and links to direct users to each README.

## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

Comments:
Hello @yeounyi,

Thank you very much for your PR, making `transformers` more user-friendly for Korean users!

Would you mind adding proper information in the [tooling](https://github.com/huggingface/transformers/blob/cde0c750af2fae6848ed3ee8be381b4f1230ecd0/utils/check_copies.py#L37-L54) to enable automated model list synchronization, so that once a new model is added into `README.md`, it will be automatically synchronized into the Korean readme.

@qqaatw I added README_ko.md in the tooling :)  

cc @sgugger
Fix assertion in models
Author: skpig
Pull request: yes

# What does this PR do?

Replace assertions in src/transformers/models/marian/convert_marian_to_pytorch.py and src/transformers/models/luke/convert_luke_original_pytorch_checkpoint_to_pytorch.py with ValueError
<!-- Remove if not applicable -->

Related to (#12789)


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?
@patrickvonplaten

Comments:
Thanks for your suggestion! @sgugger  I've changed my commit.
Mixture of non-prefixed and prefixed (B-, I-)
Author: KoichiYasuoka
Pull request: yes


# What does this PR do?

Reconsider of #13493 for mixture case, where B-TAG and I-TAG are used for multi-tokens and TAG is used for a single token.

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

@Narsil 



Comments:
This seems a bit overboard. I don't think mixed tags is desirable. (it's very uneasy to know what a model will do.
The previous fix was about being backward compatible, this is a proposed breaking change I think.

However I am not necessarily having all information.

@LysandreJik do you have an opinion ?

Pinging @stefan-it as he has extensive experience with NER, maybe you can enlighten us? :)

Yeah, I think this is also related to this issue:

https://github.com/huggingface/transformers/issues/4262

tldr: some of the NER datasets use IOB1 as labeling scheme (namely CoNLL-2003 English and German).

Hi @stefan-it , but reading the conversation it just confirms that using I- only is the way to go, and that we cannot have same but separated entity splits with this scheme. 

It seems that the proposed PR wouldn't actually solve the problem either because B- are never outputted, right ?

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.
[wav2vec2] Add missing --validation_split_percentage data arg
Author: falcaopetri
Pull request: yes

# What does this PR do?

This PR adds the missing `--validation_split_percentage` arg in wav2vec2 examples.
Default value comes from [`run_wav2vec2_pretraining_no_trainer.py`](https://github.com/huggingface/transformers/blob/70f186f61ebff4ad87fd7cc0fc1e0e0660b485ea/examples/pytorch/speech-pretraining/run_wav2vec2_pretraining_no_trainer.py#L95).

@stas00, @patrickvonplaten

Comments: