noamaanMulla-03/transformers-issues huggingface dataset:
ImportError: cannot import name 'BigBirdTokenizer' from 'transformers'
Author: zynos
Pull request: no

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 4.9.1
- Platform: windows
- Python version: 3.9
- PyTorch version (GPU?): 1.9 (CPU)
- Tensorflow version (GPU?):
- Using GPU in script?: no
- Using distributed or parallel set-up in script?:


## Information

Model I am using BigBird:

The problem arises when using:
* [ ] the official example scripts: (give details below)
* [X] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [x] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:
```

from transformers import BigBirdTokenizer,BigBirdModel

print("hello")

```
<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

## Expected behavior

No import error.
Importing **BigBirdTokenizerFast** works without a problem.

Comments:
The sentencepiece library was missing. 

`BigBirdTokenizer` requires a sentencepiece installation, but you should have had that error instead of an import error. This is because the `BigBirdTokenizer` was misplaced in the init, the PR linked above fixes it.

I sadly only got the import error, nothing else. An error indicating that sentencepiece is missing is definitely more helpful. Thanks for creating the PR

I installed sentencepiece but I got the same error:

```
!pip install --quiet sentencepiece
from transformers import BigBirdTokenizer
```
ImportError: cannot import name 'BigBirdTokenizer' from 'transformers' (/usr/local/lib/python3.7/dist-packages/transformers/__init__.py)

@MariamDundua what is the version of your transformers package?

Hi @zynos @sgugger . I'm using transformers 4.8.0 and have installed sentencepiece. But I'm having same cannot import name 'BigBirdTokenizer' issue. Thanks. 

Make sure you use the latest version of Transformers. It should include a clearer error message if the import fails.
Reproducible checkpoint
Author: sgugger
Pull request: yes

# What does this PR do?

This PR fixes reproducibility when resuming training from a checkpoint for PyTorch >= 1.6 and non-TPU trainings. The current implementation in the Trainer had two flaws:

First when going over the data at the beginning (skipping epochs and batches), the first shuffling was the same in the original training or resuming from the checkpoint, but not the subsequents shufflings. This is because those shufflings were determined by the global torch RNG which had a different state at the end of epoch 1 in a full training vs just after the shuffle of epoch 1 in the resumed training. I did not identify the exact reason but there was one or several calls to the global torch RNG during epoch 1 of the full training and when resuming we only called for the shuffle of epoch 1.

To fix this, a generator is now used to determine the shuffling, this way the generator is set with the same seed in both cases (full training or resuming from checkpoint) and then has the exact same number of calls (one shuffle per epoch) so that generator ends up in the same state in the full training or when resuming from a checkpoint.

The second thing that was different is the CUDAs RNG states, which are used in the dropout layer of the models. To fix this, all RNG states are saved when saving a checkpoint and set after all the data skip phase when resuming training.

Fixes #11504
Fixes #11323

Comments:
@stas00 Could you double check the last changes are all good with you? They include loading/saving all RNGs state (and separately in distributed training) as you suggested.

Everything looks great, @sgugger.

Given the fragile nature of this added feature it'd be relatively easily to break it, so may I suggest that at least some rudimentary test would go a long way to prevent this?

(fragile, not because how it was coded, but because it's just fragile in its own random nature)

perhaps duplicating test_trainer's resume test and generating a random number on each np, pt, py before and after resume? 

Added a new doc section in a separate PR https://github.com/huggingface/transformers/pull/11588
remove re-defination of FlaxWav2Vec2ForCTCModule
Author: patil-suraj
Pull request: yes

# What does this PR do?

Remove duplicate definations of `FlaxWav2Vec2ForCTCModule`

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_15965). All of your documentation changes will be reflected on that endpoint.

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_15965). All of your documentation changes will be reflected on that endpoint.
can't use with pytorch dataloader
Author: sipie800
Pull request: no

RuntimeError: einsum(): the number of subscripts in the equation (3) does not match the number of dimensions (4) for operand 0 and no ellipsis was given

Comments:
Please paste the whole code and describe your task in detail

@sipie800 Seems like a dimension error while resizing since you are using einsum. If you could share your code I maybe able to help.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.
Validate models are loadable
Author: borgr
Pull request: no

# 🚀 Feature request

<!-- A clear and concise description of the feature proposal.
     Please provide a link to the paper and code in case they exist. -->
Validating models uploaded to the hub in some way

## Motivation

<!-- Please outline the motivation for the proposal. Is your feature request
     related to a problem? e.g., I'm always frustrated when [...]. If this is related
     to another GitHub issue, please link here too. -->
I so far only loaded t5-small models en masse (or generally that are not the pretrained version). I have found a third of the models are unloadable!
Followed this discussion: https://discuss.huggingface.co/t/cant-download-some-models-although-they-are-in-the-hub/13944
## Your contribution

<!-- Is there any way that you could help, e.g. by submitting a PR?
     Make sure to read the CONTRIBUTING.MD readme:
     https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md -->
I am not sure, I don't know how the code to the hub itself works, I think the fix is not too complicated for others that do.
I suggest a loading code must be added to the uploading to the hub (with the default of autoWhatever.from_pretrained() or even not adding and always assuming that). Then if the users' own code fails when they try to upload, then you say upload failed unloadable. In that way there won't be models where they have a description, but actually, the files are not there, or they miss some of the files needed to load.
Moreover, this would make for an easy way for people to start, loading the model would be known (again, may be just skipped if assumed the default from_pretrained is always this way).

Comments:
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.
Update modeling_blenderbot.py
Author: labrat97
Pull request: yes

Made the `BlenderbotEncoderLayer` have similar optional arguments to the `BlenderbotDecoderLayer` and other classes.

Fixes the `forward` call of the `BlenderbotEncoderLayer`.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- albert, bert, xlm: @LysandreJik
- blenderbot, bart, marian, pegasus, encoderdecoder,  t5: @patrickvonplaten, @patil-suraj
- longformer, reformer, transfoxl, xlnet: @patrickvonplaten
- fsmt: @stas00
- funnel: @sgugger
- gpt2: @patrickvonplaten, @LysandreJik
- rag: @patrickvonplaten, @lhoestq
- tensorflow: @LysandreJik

Library:

- benchmarks: @patrickvonplaten
- deepspeed: @stas00
- ray/raytune: @richardliaw, @amogkam
- text generation: @patrickvonplaten
- tokenizers: @n1t0, @LysandreJik
- trainer: @sgugger
- pipelines: @LysandreJik

Documentation: @sgugger

HF projects:

- datasets: [different repo](https://github.com/huggingface/datasets)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Examples:

- maintained examples (not research project or legacy): @sgugger, @patil-suraj
- research_projects/bert-loses-patience: @JetRunner
- research_projects/distillation: @VictorSanh

 -->


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_15382). All of your documentation changes will be reflected on that endpoint.

Could you also update this in `modeling_mbart.py` as well, because the encoder and decoder modules are in blenderbot are copied from mbart so we need to fix both of those. So

- update the mbart file
- and run `make fix-copies`, which will then fix this for all other models which have a similar copy.

Let me know if you need help with it :) 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.
Support modern list type hints in HfArgumentParser
Author: konstantinjdobler
Pull request: yes

# What does this PR do?
Support modern list type hint syntax ([PEP 585](https://www.python.org/dev/peps/pep-0585/), introduced in Python 3.9) in the HfArgumentParser. This change keeps backwards compatibility, e.g. the old `typing.List[...]` works just as well as the newer `list[...]`.


Fixes #15950 


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

@sgugger 

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- albert, bert, xlm: @LysandreJik
- blenderbot, bart, marian, pegasus, encoderdecoder,  t5: @patrickvonplaten, @patil-suraj
- longformer, reformer, transfoxl, xlnet: @patrickvonplaten
- fsmt: @stas00
- funnel: @sgugger
- gpt2: @patrickvonplaten, @LysandreJik
- rag: @patrickvonplaten, @lhoestq
- tensorflow: @LysandreJik

Library:

- benchmarks: @patrickvonplaten
- deepspeed: @stas00
- ray/raytune: @richardliaw, @amogkam
- text generation: @patrickvonplaten
- tokenizers: @n1t0, @LysandreJik
- trainer: @sgugger
- pipelines: @LysandreJik

Documentation: @sgugger

HF projects:

- datasets: [different repo](https://github.com/huggingface/datasets)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Examples:

- maintained examples (not research project or legacy): @sgugger, @patil-suraj
- research_projects/bert-loses-patience: @JetRunner
- research_projects/distillation: @VictorSanh

 -->


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_15951). All of your documentation changes will be reflected on that endpoint.
HfArgumentParser doesn't recognize new list type hint syntax
Author: konstantinjdobler
Pull request: no

Since Python 3.9 we can use generic types as type hints (e.g. `list[str]` instead of `typing.List[str]`), see also [PEP 585](https://www.python.org/dev/peps/pep-0585/). Currently this does not seem to work with the `HfArgumentParser`, but using the old `typing.List[str]` works.

Minimal example for an arguments data class:
```python
@dataclass
class Args:
  my_boring_argument_list: typing.List[str] # works
  my_cool_argument_list: list[str] # fails
```

Comments:
I was able to pretty easily find the relevant piece of code and submitted a PR to fix it #15951
Make is_thing_map in Feature Extractor post_process_panoptic_segmentation defaults to all instances
Author: FrancescoSaverioZuppichini
Pull request: yes

# What does this PR do?

Following a discussion with @Narsil , argument `is_thing_map` in `FeatureExtractor.post_process_panoptic_segmentation` will default to consider all instances `thing`. Thus, it won't perform instances merging. 

The user should always provide a correct `is_thing_map` if he wants to merge instances, currently, the code will default to `COCO` that may be unwanted. 

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_15954). All of your documentation changes will be reflected on that endpoint.

> Not particularly fond of the names `thing` vs `stuff` but I imagine this is the standards ?

More or less :) 


Thanks @Narsil  for the very nice feedback. Following your amazing set of examples, I believe `class_ids_to_fuse` (or `label_ids_to_fuse`) is the most descriptive name we can use. 

As you said, a `Dict` is not necessary. My implementation followed more or less what was done by the authors, but we can improve it. 

I'll update the code.
Update training scripts docs
Author: stevhliu
Pull request: yes
Labels: name: Documentation

This PR updates the [Examples](https://huggingface.co/docs/transformers/examples) docs with examples for running a training script on distributed setups, with mixed precision, TPUs and Accelerate. It also adds examples for helpful options like using your own custom dataset, resume training from a checkpoint, and uploading to the Hub. 

The examples are centered around one task, but the user is expected to be able to generalize this guide to training scripts for other tasks as well. Feel free to let me know if I'm missing any major difference between the tasks that should be mentioned. I also created this guide as a new `mdx` file to be able to use the framework switcher, so we can remove `examples` from the toctree when we merge.

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_15931). All of your documentation changes will be reflected on that endpoint.
Set scale_embedding to False in some TF tests
Author: ydshieh
Pull request: yes

# What does this PR do?

This PR set `scale_embedding=False` in `TFSpeech2TextModelTester` to avoid `inputs_embeds` and the PT/TF difference being scaled by 4 - the objective is to keep a low tolerance `1e-5` in the PT/TF test, as we see several times this is a strong safe guard!

(This is not a real bug. It's also similar to #15684, where we got larger differences between PT/TF simply because the model weights are initialized with larger values).

TF: @gante @Rocketknight1 
Speech: @patrickvonplaten 

## More context

Set `scale_embedding` to `False` in some TF tests.

Current `Speech2TextConfig` has default [scale_embedding=True](https://github.com/huggingface/transformers/blob/9932ee4b4bca9045d941af6687ef69eedcf68483/src/transformers/models/speech_to_text/configuration_speech_to_text.py#L137). Therefore we have

- [self.embed_scale = tf.math.sqrt(float(embed_dim))](https://github.com/huggingface/transformers/blob/9932ee4b4bca9045d941af6687ef69eedcf68483/src/transformers/models/speech_to_text/modeling_tf_speech_to_text.py#L742).
- The tests for `speech_to_text` has [hidden_size=16](https://github.com/huggingface/transformers/blob/9932ee4b4bca9045d941af6687ef69eedcf68483/tests/speech_to_text/test_modeling_tf_speech_to_text.py#L75), and therefore `inputs_embeds` will be scaled by 4.

https://github.com/huggingface/transformers/blob/9932ee4b4bca9045d941af6687ef69eedcf68483/src/transformers/models/speech_to_text/modeling_tf_speech_to_text.py#L843-L844

Since `inputs_embeds` here is obtained by some (conv.) layer instead of via look-up table, it contains some tiny difference between PT/TF. This difference is scaled by 4 through `self.embed_scale`.

This makes `TFSpeech2TextModel` **the only one** model that will fail the aggressive PT/TF test introduced in #15839 (with low tolerance `1e-5`). More precisely, the output tensors failed are `encoder_hidden_states_0` and `encoder_hidden_states_1`.

## Results

**With this PR, the tolerance `1e-5` works for all TF models' `test_pt_tf_model_equivalence` (in #15839), both on GPU / CPU, tested 100 times per model.**

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_15952). All of your documentation changes will be reflected on that endpoint.
How to use GPT-2 for predicting the next word in batch
Author: etsurin
Pull request: no

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version:
- Platform:transformers 4.16.0.ver0
- Python version:3.7
- PyTorch version (GPU?):1.10.1
- Tensorflow version (GPU?):
- Using GPU in script?:yes
- Using distributed or parallel set-up in script?:no

### Who can help
<!-- Your issue will be replied to more quickly if you can figure out the right person to tag with @
 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.-->
@patrickvonplaten @LysandreJik 

### Information


I'm using GPT2 for predicting the next word and I expect to get the loss. For running the model in batch, I should make paddings so that all the samples are of the same length. But when padding I got the significantly different loss values.

```
from transformers import GPT2LMHeadModel, GPT2Tokenizer
import torch
import numpy as np

device = 'cuda'
model_id = 'gpt2-medium'
model = GPT2LMHeadModel.from_pretrained(model_id).to(device)
model.eval()
tokenizer = GPT2Tokenizer.from_pretrained(model_id)
def get_loss(input,target,encoded=True):
    input=input.to(device)
    target=torch.tensor(target).to(device)
    #print(final_input,target)
    output=model(**input,return_dict=True)
    output = output.logits[0][-1:]
    print(output)
    loss_fct = torch.nn.CrossEntropyLoss()
    #print(output.view(-1, output.size(-1)).shape, target.shape)
    loss = loss_fct(output.view(-1, output.size(-1)), target)
    return loss.cpu().detach()

tokenizer.pad_token = '!'
tokenizer.padding_side = 'left'
text='I like playing the'
target=' piano'
target=tokenizer.encode(target)
text1=tokenizer(text, return_tensors="pt")
print(text1)
text2=tokenizer(text,padding='max_length',max_length=300, return_tensors="pt")
print(text2)
print(get_loss(text1,target))
print(get_loss(text2,target))
```

And I got the following results:

#### not padding

```
tensor([[-67.2690, -65.8415, -70.7127,  ..., -70.0382, -72.1321, -67.0170]],
       device='cuda:0', grad_fn=<SliceBackward>)
tensor(4.4344)
```

#### padding

```
tensor([[-205.1836, -207.1315, -211.1725,  ..., -218.3117, -216.3883,
         -205.4879]], device='cuda:0', grad_fn=<SliceBackward>)
tensor(16.0859)
```

The loss is much higher after padding. And the logits value is also strange. 
I have to pad on the left to make sure logits[-1:]  is always the encode state of the last input. 

## Expected behavior

I think that GPT2 can do this task but I don't know how to use it in batch. 
 
<!-- A clear and concise description of what you would expect to happen. -->


Comments:
Add custom classifcation dataset - computer vision
Author: rozeappletree
Pull request: no

Hi, I recently read your amazing blog https://huggingface.co/blog/fine-tune-vit

There I could not find any reference to how to create our own dataset. Is there any class helper that we can use? Something like -

```python
from xyz import ImageClassificationDataset

dataset = ImageClassificationDataset.from_directory('/path/to/dataset')
```

For directory structured as

```
train
  +- class0
  +- class1
  ...
  +- classn
valid
  +- class0
  +- class1
  ...
  +- classn
test
  +- class0
  +- class1
  ...
  +- classn
```
where every class folder has images



Comments:
Or, If you don't have that kind of class what would be the best way to generate dataset dict from folder structure mentioned above
```python
DatasetDict({
    train: Dataset({
        features: ['image_file_path', 'image', 'labels'],
        num_rows: 1034
    })
    validation: Dataset({
        features: ['image_file_path', 'image', 'labels'],
        num_rows: 133
    })
    test: Dataset({
        features: ['image_file_path', 'image', 'labels'],
        num_rows: 128
    })
})
```

Hi,

We've just added the [ImageFolder](https://huggingface.co/docs/datasets/master/en/loading#image-folders) to the Datasets library :)

This way, you can easily load your custom data as follows:

```
from datasets import load_dataset

dataset = load_dataset("imagefolder", data_dir="/path/to/data")
```
Fix LayoutLMv2 test
Author: NielsRogge
Pull request: yes

# What does this PR do?

This PR fixes a failing LayoutLMv2 test, `test_torch_encode_plus_sent_to_model`.

It also removes `requires_scatter`, because LayoutLMv2 doesn't depend on it.

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_15939). All of your documentation changes will be reflected on that endpoint.
[Tests] Fix ViTMAE integration test
Author: NielsRogge
Pull request: yes

# What does this PR do?

This PR fixes the integration test of ViTMAE, which only passed on CPU.

ViTMAE uses randomness inside (it creates a random boolean mask to indicate which patches to mask). I used `torch.manual_seed(2)` to make this deterministic, however I [learned](https://discuss.pytorch.org/t/random-seed-that-spans-across-devices/19735) that using the same seed across CPU/GPU will not result in the same random numbers (as both use a different random number generator).

Hence, this PR sets the expected slice depending on the device.

Huge thanks to @ydshieh for helping me out finding this.

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_15949). All of your documentation changes will be reflected on that endpoint.