noamaanMulla-03/transformers-issues huggingface dataset:
Added a varification step for to the development contribution guide 
Author: CakeCrusher
Pull request: yes

# What does this PR do?
Informs the user to have the proper python version to ensure they can get a development environment working.

Fixes #18112


## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [x] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

@sgugger 


Comments:
_The documentation is not available anymore as the PR was closed or merged._

closing due to unrelated code
TF: `unpack_inputs` decorator independent from `main_input_name`
Author: gante
Pull request: yes

# What does this PR do?

As the title indicates -- the `unpack_inputs` decorator becomes independent from `main_input_name` in this PR.

The old `input_processing` included some checks for `input_ids`, which somewhat transitioned into the `unpack_inputs` decorator (the `input_ids` input was obtained there from the argument under `main_input_name`). However, in practice, it is not needed -- what we want is to support the case where all model arguments come packed in the first input, which happens to be `main_input_name` for most use cases of `unpack_inputs`. Note that Keras often expects this packing behavior with input dictionaries, which `input_processing` maps back to our expected format.

Fixes #18040

Comments:
_The documentation is not available anymore as the PR was closed or merged._
supported python versions reference
Author: CakeCrusher
Pull request: yes

# What does this PR do?

Provides a reference to the supported python versions to get a development environment working.

Fixes #18112


## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [x] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?
@sgugger 


Comments:
_The documentation is not available anymore as the PR was closed or merged._

@sgugger another thing to consider is that we are only referencing the line number so the moment the file is updated, and the lines shift it will link to something else. Any fixes come to mind? Or will that do for the time being?
[Pipeline] avoid importing tensorflow if not used
Author: nmntz
Pull request: yes

# What does this PR do?

Avoids loading unnecessary modules by `pipelines.base.infer_framework_load_model()` which could create some unexpected behaviour like tensorflow allocating all GPU memory.
@sgugger @LysandreJik 

Before:
```python
from transformers import pipeline
pipeline("text-classification")
# This would try importing `TFDistilBertForSequenceClassification`if both tensorflow and pytorch
# are available, and tensorflow would allocate all GPU memory, even if we expect to use
# the pytorch model
```

After:

```python
from transformers import pipeline
pipeline("text-classification")
# Only `DistilBertForSequenceClassification` is imported, and tensorflow is not called
```


Comments:
_The documentation is not available anymore as the PR was closed or merged._

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

Was this issue resolved in another PR already? @sgugger 

The fact that TensorFlow takes all GPU memory has been fixed in #18044
Make sharded checkpoints work in offline mode
Author: sgugger
Pull request: yes

# What does this PR do?

This PR make sharded checkpoint work in offline mode and add more information to an error we return.
The crux of the issue is that the `from_pretrained` method of the various models will catch `EntryNotFoundError` on the regular model weights file, but we return a `FileNotFoundError` in offline mode. I changed the error type at the root, to avoid making three modifications in the PyTorch/TF/Flax model classes, but can change if you don't find this suitable.

Comments:
_The documentation is not available anymore as the PR was closed or merged._
Add TF DeiT implementation
Author: amyeroberts
Pull request: yes

# What does this PR do?

Adds the TF implementations of DeiT

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [x] Did you write any new necessary tests?

Comments:
_The documentation is not available anymore as the PR was closed or merged._

Please also incorporate the updates of #17731 

> Please also incorporate the updates of https://github.com/huggingface/transformers/pull/17731

@NielsRogge Will do! I originally added, but it resulted in lots of changes because of all the `Copied From` statements, so will wait until your PR is merged and those updates finalised. 

@sgugger could you maybe give it a quick review as it's vision? (otherwise happy to do it if you're busy)
how can I use emformer checkpoint?
Author: dykim3
Pull request: no

### Feature request

ImportError                               Traceback (most recent call last)
<ipython-input-1-7a40e4bd817f> in <module>
----> 1 from transformers import EmformerForRNNT
      2 
      3 model = EmformerForRNNT.from_pretrained("anton-l/emformer-base-librispeech")

ImportError: cannot import name 'EmformerForRNNT' from 'transformers' (  .local/lib/python3.8/site-packages/transformers/__init__.py)

### Motivation

.

### Your contribution

.

Comments:
Hi,

Emformer hasn't been added to the library yet.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
About `do_basic_tokenize` behavior in BertTokenizer
Author: GuillemGSubies
Pull request: no
Labels: name: wontfix

name: Documentation

# ❓ Questions & Help

In the [docs](https://huggingface.co/transformers/model_doc/bert.html?highlight=basic%20tokenization#berttokenizer) it just says that it performs a basic tokenization before the word piecer. But what is actually a "basic tokenization"? I would really appreciate a little more of information on that.

Diving into the code I found out that the basic tokenization removes control characters from the text. I did not expect that behavior from what I read on the docs. That gave us problems because some characters like `` weren't tokenizing and we didn't know why.

More generally I think that transformers has a problem with the docs. Most of the times that something does not work for me, I don't bother to look at them and I directly dive into the source code in order to understand what's happening. I am used to scikit-learn and maybe I am a bit bisaed, but I really think that these kind of things can be a barrier for new people wanting to use transformers.

If there is something I can do to help, I am happy to send a PR.

Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


there is a `BasicTokenizer` in previous `pytorch_transformers` package:
![image](https://user-images.githubusercontent.com/24619551/178862459-31bdbb45-abda-4e36-a94d-a85b89734604.png)
This must be what "basic tokenize" does
LayoutLMv3 not downloading via official code samples
Author: microcoder-py
Pull request: no
Labels: name: bug

### System Info

```shell
Used the official code sample for the microsoft/layoutlmv3-base model, but it is not working

Link to code: https://huggingface.co/docs/transformers/main/en/model_doc/layoutlmv3#transformers.LayoutLMv3Model

Error: 


KeyError                                  Traceback (most recent call last)
<ipython-input-17-2e4d79cdf031> in <module>()
      3 import torch
      4 
----> 5 processor = AutoProcessor.from_pretrained("microsoft/layoutlmv3-base")
      6 model = AutoModelForSequenceClassification.from_pretrained("microsoft/layoutlmv3-base")
      7 

2 frames
/usr/local/lib/python3.7/dist-packages/transformers/models/auto/processing_auto.py in from_pretrained(cls, pretrained_model_name_or_path, **kwargs)
    198             )
    199             if tokenizer_config_file is not None:
--> 200                 with open(tokenizer_config_file, encoding="utf-8") as reader:
    201                     config_dict = json.load(reader)
    202 

/usr/local/lib/python3.7/dist-packages/transformers/models/auto/configuration_auto.py in from_pretrained(cls, pretrained_model_name_or_path, **kwargs)
    698                     " set the option `trust_remote_code=True` to remove this error."
    699                 )
--> 700             if kwargs.get("revision", None) is None:
    701                 logger.warning(
    702                     "Explicitly passing a `revision` is encouraged when loading a configuration with custom code to "

/usr/local/lib/python3.7/dist-packages/transformers/models/auto/configuration_auto.py in __getitem__(self, key)
    407     A dictionary that lazily load its values when they are requested.
    408     """
--> 409 
    410     def __init__(self, mapping):
    411         self._mapping = mapping

KeyError: 'layoutlmv3'
```
```


### Who can help?

@Lysan

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

https://colab.research.google.com/drive/11aIci0c_UuId5BK2-U6QwZgEKE3ran9I?usp=sharing

### Expected behavior

```shell
The model downloads and executes with the same behaviour as described on HF
```


Comments:
I believe LayoutLM-v3 is not in an official release yet, so you'll need to install from source in order to use it for now.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Add type hints for XLM TensorFlow
Author: elusenji
Pull request: yes

# What does this PR do?
This PR adds type hints to the XLM model for TensorFlow.

@LysandreJik 


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_16658). All of your documentation changes will be reflected on that endpoint.

@elusenji you might have unintentionally added commits from other contributors. You would want to remove other's changes.

@kumar-abhishek does that fix the issue you highlighted?

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

Is this still being reviewed? All the checks were successfully passed.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

Sorry, this PR completely flew over the radar! Thanks for working on it, and please ping @Rocketknight1 for the type hints PRs :)

@elusenji It looks good now, but unfortunately, things moved on since the PR was filed and now we have a bunch of new errors! Can you pull changes from our repo to your fork, and then rebase your branch? That should hopefully resolve these errors, and then this PR should be ready to merge.

> @elusenji It looks good now, but unfortunately, things moved on since the PR was filed and now we have a bunch of new errors! Can you pull changes from our repo to your fork, and then rebase your branch? That should hopefully resolve these errors, and then this PR should be ready to merge.

Done, but not sure if I did this correctly.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
[TRACKER] Add alibi tests on BLOOM
Author: younesbelkada
Pull request: no
Labels: name: WIP

### Feature request

Add ALiBi tests on BLOOM ! 
We should add several tests to simply to test if alibi has been created correctly

- test padding 
- test expected output

@Narsil 

### Motivation

Build stronger CI tests

### Your contribution

Design and build the tests mentioned above

Comments:
Pipelines: batch size
Author: ioana-blue
Pull request: no

I'm using a pipeline with feature extraction and I'm guessing (based on the fact that it runs fine on the cpu but dies with out of memory on gpu) that the `batch_size` parameter that I pass in is ignored. Can pipeline be used with a batch size and what's the right parameter to use for that?

This is how I use the feature extraction:
```
# use pipelines and feature extraction
    feature_extractor = pipeline(
            task="feature-extraction",
            model=model_args.model_name_or_path,
            config = config,
            tokenizer = tokenizer,
            framework="pt",
            batch_size=data_args.batch_size,
            truncation=True,
        )

....
outputs = feature_extractor(inputs = predict_inputs, truncation=True)
```

@Narsil has been really helpful with pipelines, perhaps he knows the answer?


Comments:
Hi,

What's your batch size ?

Do you mind sharing the version of transformers you are running ?
Also what's the model and the predict inputs ?

Batch_size is implemented for this pipeline, getting OOM, means probably that the batch_size is just too big, try setting it at 1 first probably to check if that fixes the issue.

Ideally when you share such an issue if you can provide a reproducible script + an actual error output it helps us tremendously in diagnosing our issue.

Cheers

I used 16 as the batch size, I installed directly from git yesterday so pretty up to date. When I manually split in the same batch size (so really send the inputs in batches of 16 from the start to the feature extractor), it works, this is what made me suspect that batching may not work. 

There is not much for the error, the job dies with the gpu spitting an OOM error. 

I also noticed another performance issue and I have a snippet of the script there where I manually batched the samples. See #14333 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.

I had a similar issue where the batch_size wasn't changing regardless of what argument I entered. I fixed this by changing the batch and batch_eval values in the config object that you pass into  input_pipeline.get_data_from_tfds()

Hi @CallumJMac ,

batching doesn't work with TF (yet). I was convinced a warning was in place but I fail to see it now. 

Could that be the issue ?
Also could you share your solution, it could help setting up automated batching on the pipelines for TF. If you're willing to do a PR on it, I would be glad to help too !

Cheers.
core dumped
Author: TomasAndersonFang
Pull request: no

### System Info

```shell
centos7
transformers==4.17.0
datasets==1.17.0
pytorch==1.6.0
GPUs==4*GTX 3090
```


### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

I modify the example code of text classification (run_glue.py) and train it on my own dataset. When I run this code on the CPU, it is well running (setting no_cuda true). But when I run it on GPUs, it quits when finishing loading data and generate the following information:

run_burebert_ft.sh: line 16: 37055 Aborted                 (core dumped) CUDA_LAUNCH_BLOCKING=1 python run_br_pred.py --model_name_or_path ../plm/BureBERT --train_file ../dataset/priority_pred_data/priority_train.csv --validation_file ../dataset/priority_pred_data/priority_valid.csv --test_file ../dataset/priority_pred_data/priority_test.csv --cache_dir ./cache_dir --do_train --do_eval --do_predict --no_cuda false --max_seq_length 128 --per_device_train_batch_size 32 --learning_rate 5e-6 --num_train_epochs 10 --save_steps 10000 --output_dir ./results_priority

I am very confused because it just gives limited information. Actually, I also run the same code on another server which contains one tesla V100 GPU, it can run well. So I guest whether It requires extra settings when training on multiple GPUs.

### Expected behavior

```shell
When I run the code, it should start to fine-tune BureBERT on my dataset for ten epochs.
```


### Checklist

- [X] I have read the migration guide in the readme. ([pytorch-transformers](https://github.com/huggingface/transformers#migrating-from-pytorch-transformers-to-transformers); [pytorch-pretrained-bert](https://github.com/huggingface/transformers#migrating-from-pytorch-pretrained-bert-to-transformers))
- [X] I checked if a related official extension example runs on my machine.

Comments:
I find that my problem is produced by cuda version. It seems that GTX 3090 is not compatible with cuda 10 and torch installed by pip has the default cuda version (10). So I re-install pytorch with cuda 11 and solve the problem.
LongT5 Summarization Example Not Working
Author: ghost
Pull request: no
Labels: name: bug

### System Info

- OS: Ubuntu 20.04.4 LTS focal
- Conda: 4.12.0
- Python: 3.7.14
- Pip: 22.1.2
- Torch: 1.12.0
- Transfromers: 4.16.2
- NVIDIA-SMI: 510.73.05
- Nvcc -V: cuda V11.3.109

### Who can help?

@patrickvonplaten, @ydshieh, @sgugger 

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

`git clone --branch v4.16.2-release https://github.com/huggingface/transformers`

Example from [Transformers/examples/pytorch/summarization](https://github.com/huggingface/transformers/tree/main/examples/pytorch/summarization), the only change is the `--model_name_or_path`
```
python transformers/examples/pytorch/summarization/run_summarization.py \
    --model_name_or_path google/long-t5-tglobal-base \
    --do_train \
    --do_eval \
    --dataset_name cnn_dailymail \
    --dataset_config "3.0.0" \
    --source_prefix "summarize: " \
    --output_dir /tmp/tst-summarization \
    --per_device_train_batch_size=4 \
    --per_device_eval_batch_size=4 \
    --overwrite_output_dir \
    --predict_with_generate
```
**Error:**
```
[INFO|configuration_utils.py:644] 2022-07-16 13:23:14,077 >> loading configuration file https://huggingface.co/google/long-t5-tglobal-base/resolve/main/config.json from cache at /home/good/.cache/huggingface/transformers/1b9067139467923bb0ea7749ceb5694acb0950b479ad1ebe47d9014180af8c31.69c5bfb92a1a084ead5ef0d9c9c9f09bac4f07cfd875433aa8fab59199208a7f
Traceback (most recent call last):
  File "transformers/examples/pytorch/summarization/run_summarization.py", line 698, in <module>
    main()
  File "transformers/examples/pytorch/summarization/run_summarization.py", line 371, in main
    use_auth_token=True if model_args.use_auth_token else None,
  File "/home/good/anaconda3/envs/gpt1/lib/python3.7/site-packages/transformers/models/auto/configuration_auto.py", line 632, in from_pretrained
    config_class = CONFIG_MAPPING[config_dict["model_type"]]
  File "/home/good/anaconda3/envs/gpt1/lib/python3.7/site-packages/transformers/models/auto/configuration_auto.py", line 347, in __getitem__
    raise KeyError(key)
KeyError: 'longt5'
```

### Expected behavior

LongT5 model from [google/long-t5-tglobal-base](https://huggingface.co/google/long-t5-tglobal-base) to start training like a normal T5 model ([T5-base](https://huggingface.co/t5-base)).

Comments:
update transformer to 4.20.0+ should solve this issue.

Also I don't think you need --source_prefix "summarize: "  according to the paper.

@whaleloops It works, thanks.