noamaanMulla-03/transformers-issues huggingface dataset:
Remove Bert tokenizer dependency from DistillBert (slow/fast) tokenizers
Author: IvanLauLinTiong
Pull request: yes

Hi @sgugger,

Fixes https://github.com/huggingface/transformers/issues/19303
- The `BertTokenizer` dependency has been removed from `DistillBerTokenizer`
- The `BertTokenizerFast` dependency has been removed from `DistillBerTokenizerFast`

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?
@sgugger 

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- text models: @ArthurZucker and @younesbelkada
- vision models: @amyeroberts and @NielsRogge
- speech models: @sanchit-gandhi

Library:

- flax: @sanchit-gandhi
- generate: @gante
- pipelines: @Narsil
- tensorflow: @gante and @Rocketknight1
- tokenizers: @ArthurZucker
- trainer: @sgugger

Integrations:

- deepspeed: @stas00
- ray/raytune: @richardliaw, @amogkam

Documentation: @sgugger and @stevhliu

HF projects:

- accelerate: [different repo](https://github.com/huggingface/accelerate)
- datasets: [different repo](https://github.com/huggingface/datasets)
- diffusers: [different repo](https://github.com/huggingface/diffusers)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Maintained examples (not research project or legacy):

- Flax: @sanchit-gandhi
- PyTorch: @sgugger
- TensorFlow: @Rocketknight1

 -->


Comments:
_The documentation is not available anymore as the PR was closed or merged._
Adds type checking to PreTrainedConfig.
Author: mmcdermott
Pull request: yes

# What does this PR do?

Fixes [# (issue)](https://github.com/huggingface/transformers/issues/20915)


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [X] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [X] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [X] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- text models: @ArthurZucker and @younesbelkada
- vision models: @amyeroberts and @NielsRogge
- speech models: @sanchit-gandhi

Library:

- flax: @sanchit-gandhi
- generate: @gante
- pipelines: @Narsil
- tensorflow: @gante and @Rocketknight1
- tokenizers: @ArthurZucker
- trainer: @sgugger

Integrations:

- deepspeed: @stas00
- ray/raytune: @richardliaw, @amogkam

Documentation: @sgugger and @stevhliu

HF projects:

- accelerate: [different repo](https://github.com/huggingface/accelerate)
- datasets: [different repo](https://github.com/huggingface/datasets)
- diffusers: [different repo](https://github.com/huggingface/diffusers)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Maintained examples (not research project or legacy):

- Flax: @sanchit-gandhi
- PyTorch: @sgugger
- TensorFlow: @Rocketknight1

 -->


Comments:
_The documentation is not available anymore as the PR was closed or merged._

Mmm, the tests are all failing for weird reasons. It seems the branch you are using for the PR is pretty outdated compared to main. Could you do a quick rebase?

Huh... sorry, I thought I corrected that before pushing. I'll do a rebase and get things squared away. 

Thanks a lot!
Fixing DistilBert error message
Author: samuelzxu
Pull request: yes

# What does this PR do?

Distilbert fix from [this thread](https://github.com/huggingface/transformers/pull/20933/commits/6f0282dd13d646b0f58d99a0c19377646efc2d55#r1059242097)


## Before submitting
- [X] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [X] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

@sgugger 

Comments:
_The documentation is not available anymore as the PR was closed or merged._
Finetunig of wav2vec2-xls-r-300m outputs invalid words for Bengali data
Author: manjuke
Pull request: no

### System Info

I have used wav2vec2 pretrained model of wav2vec2-xls-r-300m, and finetuned it to 1000hrs Bengali dataset. Training took 4 full days with 20 epochs. But, there is issue in decoding.  It is decoding in some arbitrary fashion, basically outputs random combination of Bengali letters (which does not have any meaning as confirmed by Bengali natives). It is showing a WER of 100% for all the sentences.

My code is based on the notebook at  https://colab.research.google.com/github/patrickvonplaten/notebooks/blob/master/Fine_Tune_XLS_R_on_Common_Voice.ipynb
@patrickvonplaten, @anton-l @sanchit-gandhi Pls suggest on what could have gone wrong. Should I use fairseq & redo the experiments?
Thanks
Eval loss graph on tensorboard looks like below.
![image](https://user-images.githubusercontent.com/6142443/203315605-d0453531-b0f6-47ad-8e6c-f04c94409938.png)

### Who can help?

@patrickvonplaten, @anton-l, @sanchit-gandhi 

### Information

- [X] The official example scripts
- [X] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

Finetuning on XLSR 300m models using Bengali dataset resulted this behaviour

### Expected behavior

WER should have been less than 100%, and should have outputted reasonably readable hypothesis words

Comments:
Hey @manjuke! Cool to see you're using XLS-R for fine-tuning on Bengali. The "issues" page on Transformers is reserved for issues related to the Transformers modelling code. For questions related to fine-tuning experiments, the forum is the best place to ask: https://discuss.huggingface.co

Could you copy your question over there? If you tag me (@sanchit-gandhi) I'll aim to answer quickly! Could you also provide a script / colab / command to reproduce this behaviour? From your eval/loss curve, it looks like you're overfitting quite drastically on your training set. This shouldn't happen for a dataset with 1k training hours, so something is definitely up!

Sure I have created a new ticket @ https://discuss.huggingface.co/t/finetunig-of-wav2vec2-xls-r-300m-outputs-invalid-words-for-bengali-data/26507

@sanchit-gandhi Could you please reply to my issue raised on hugging face. Awaiting response. Thanks

Replied on the forum!

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
PreTrainedModel: provide a more intuitive way of getting the current size of embeddings
Author: damian0815
Pull request: no

### Feature request

Please add an intuitive and obvious method to get the size of the current token embeddings on `PreTrainedModel`.

For example, a method called `get_token_embeddings_size()` that can be used as a complement to `resize_token_embeddings()`.

### Motivation

Current API design around the `resize_token_embeddings()` method requires doing the following to increase the size of the token embeddings by 1:

```python
# add 1 new embedding
current_embeddings = my_xformer.resize_token_embeddings(None)
new_embeddings_size = current_embeddings.num_embeddings + 1
my_xformer.resize_token_embeddings(new_embeddings_size)
```

This is counterintuitive and bleeds implementation details to the call site. It requires me to know
1. that calling a "resize" method with the argument `None` returns the object to be resized (which is not intuitive), and
2. that "size" means the property `num_embeddings` on the returned object (admittedly it's not difficult to guess, but it is still a *guess*, and is in fact an implementation detail that I shouldn't need to know).

It would be better if I could do something like this:

```python
# add 1 new embedding
current_embeddings_size = my_xformer.get_token_embeddings_size()
new_embeddings_size = current_embeddings_size + 1
my_xformer.resize_token_embeddings(new_embeddings_size)
```

This provides an intuitively-named method to determine the current "size", and appropriately hides the implementation detail that "size" means the `num_embeddings` property on the object to be resized.

### Your contribution

Here's a proposed implementation:

```python
    def get_token_embeddings_size(self) -> int:
        return model_embeds.num_embeddings
```

i can make a PR if it would help.

Comments:
Thanks for flagging, I had kind of the same problem recently when implementing #20043. I used
```
model.get_input_embeddings().weight.shape[0]
```
to get the embedding size, not sure if there is anything easier but having a method that does this would certainly be more helpful if you want to contribute it!


PR made

I don't think you actually opened it, you just created a new branch :-)

ok PR actually made :)

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
The Document of Pipelines are not complete.
Author: zhaowei-wang-nlp
Pull request: no

### System Info

transformers 4.24.0

### Who can help?

@sgugger @stevhliu 

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

https://huggingface.co/docs/transformers/main/en/main_classes/pipelines#transformers.FillMaskPipeline.__call__
The "\_\_call\_\_" function only describes what will happen when I pass a list of samples into the pipeline. However, this function also receives other input, such as a dataset: https://huggingface.co/docs/transformers/main/en/main_classes/pipelines#pipeline-batching

The document currently doesn't have a description of what will happen if I pass a dataset to the pipeline. This bothers me a lot. For example, when I pass a dataset and use batch_size=64 in this pipeline, I expect that I will get the results of 64 samples. However, the pipeline only returns the result of the next sample and I need to iterate 64 times to get the results of the whole batch. This doesn't involve any fatal error but is very confusing.

### Expected behavior

add a description of passing a dataset to a pipeline.

Comments:
cc @Narsil 

Is this doc more helpful ?

https://huggingface.co/docs/transformers/v4.24.0/en/main_classes/pipelines

We definitely need to update the docs for pipeline overall.



This doc still doesn't mention that results will be returned one by one (instead of in a batch) if I pass a dataset to the pipeline.

https://github.com/huggingface/transformers/pull/20437

Don't hesitate to comment on the PR if you feel it's not clear enough.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
[trainer: `distributed_concat`] ensure `all_gather`'s inputs are contiguous
Author: stas00
Pull request: yes

This PR fixes https://github.com/huggingface/transformers/issues/20942 where a user's code results in a non-contiguous tensor being passed to `all_gather` which fails with:

```
Traceback (most recent call last):
  File "contiguous.py", line 83, in <module>
    preds = torch.tensor(trainer.predict(eval_dataset)[0])
  File "/mnt/nvme0/code/huggingface/transformers-master-2/src/transformers/trainer.py", line 2894, in predict
    output = eval_loop(
  File "/mnt/nvme0/code/huggingface/transformers-master-2/src/transformers/trainer.py", line 3024, in evaluation_loop
    logits = self._nested_gather(logits)
  File "/mnt/nvme0/code/huggingface/transformers-master-2/src/transformers/trainer.py", line 3140, in _nested_gather
    tensors = distributed_concat(tensors)
  File "/mnt/nvme0/code/huggingface/transformers-master-2/src/transformers/trainer_pt_utils.py", line 191, in distributed_concat
    return type(tensor)(distributed_concat(t, num_total_examples) for t in tensor)
  File "/mnt/nvme0/code/huggingface/transformers-master-2/src/transformers/trainer_pt_utils.py", line 191, in <genexpr>
    return type(tensor)(distributed_concat(t, num_total_examples) for t in tensor)
  File "/mnt/nvme0/code/huggingface/transformers-master-2/src/transformers/trainer_pt_utils.py", line 191, in distributed_concat
    return type(tensor)(distributed_concat(t, num_total_examples) for t in tensor)
  File "/mnt/nvme0/code/huggingface/transformers-master-2/src/transformers/trainer_pt_utils.py", line 191, in <genexpr>
    return type(tensor)(distributed_concat(t, num_total_examples) for t in tensor)
  File "/mnt/nvme0/code/huggingface/transformers-master-2/src/transformers/trainer_pt_utils.py", line 191, in distributed_concat
    return type(tensor)(distributed_concat(t, num_total_examples) for t in tensor)
  File "/mnt/nvme0/code/huggingface/transformers-master-2/src/transformers/trainer_pt_utils.py", line 191, in <genexpr>
    return type(tensor)(distributed_concat(t, num_total_examples) for t in tensor)
  File "/mnt/nvme0/code/huggingface/transformers-master-2/src/transformers/trainer_pt_utils.py", line 194, in distributed_concat
    dist.all_gather(output_tensors, tensor)
  File "/home/stas/anaconda3/envs/py38-pt113/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py", line 2275, in all_gather
    work = default_pg.allgather([tensor_list], [tensor])
RuntimeError: Tensors must be contiguous
```

the fix adds `.contiguous()` which will do nothing if the tensor is already contiguous and will make it contiguous if it is not.


Fixes: https://github.com/huggingface/transformers/issues/20942

Comments:
_The documentation is not available anymore as the PR was closed or merged._
Hugging face community on fb group 
Author: Mohammed20201991
Pull request: no

Hello everyone you can  join our group to discuss new state-of-the-art methods in deep learning, machine learning, and natural language processing through the  below :
[link](https://fb.me/g/p_Qe5xCLF7zxgu4sYn/NRWMjdeM)


Comments:
Hey there! Thanks a lot for creating and sharing the group :fire: 
FYI, there is also a community Discord server with over 16000 members. You can join in [hf.co/join/discord](http://hf.co/join/discord) :hugs: 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Link to Google Colab notebook not working
Author: jaimebw
Pull request: no

Hi, it seems that the link to ```A notebook on how to [finetune BART for summarization with fastai using blurr](https://colab.research.google.com/github/ohmeow/ohmeow_website/blob/master/_notebooks/2020-05-23-text-generation-with-blurr.ipynb).``` is broken.  
@patrickvonplaten 

Comments:
cc @ohmeow who wrote the notebook.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Questions on ViT and ViT-MAE model image preprocessing
Author: zhoutong-fu
Pull request: no

### System Info

Latest version

### Who can help?

@NielsRogge

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

Hi @NielsRogge, I'm reading the ViT image processing from HF and its original implementation and have a few questions on image processing. Really appreciate your helping me understand the differences.

**ViT**
The JAX implementation from Google rescales the pixel range to [-1, 1]: https://github.com/google-research/vision_transformer/blob/main/vit_jax/input_pipeline.py#L214
HF rescales it by a factor of 1/255: https://github.com/huggingface/transformers/blob/main/src/transformers/models/vit/image_processing_vit.py#L80

**ViT MAE**
PyTorch implementation from Meta resizes the image with PIL.Image.BICUBIC interpolation: https://github.com/facebookresearch/mae/blob/efb2a8062c206524e35e47d04501ed4f544c0ae8/util/datasets.py#L59

HF uses BILINEAR https://github.com/huggingface/transformers/blob/main/src/transformers/models/vit/image_processing_vit.py#L78

### Expected behavior

I'd like to have clarifications of the questions above.

Comments:
Hi,

Thanks for your interest in ViT and ViT MAE!

Regarding point 1; ViT was ported from the timm library, which also uses a scale factor of 1/255. This can be verified as follows:

```
from timm.data import resolve_data_config
from timm.data.transforms_factory import create_transform

model = timm.create_model('vit_base_patch16_224',pretrained=True)
model.eval()

# Create Transform
transform = create_transform(**resolve_data_config(model.pretrained_cfg, model=model))
print(transform)
```
which prints:
```
Compose(
    Resize(size=248, interpolation=bicubic, max_size=None, antialias=None)
    CenterCrop(size=(224, 224))
    ToTensor()
    Normalize(mean=tensor([0.5000, 0.5000, 0.5000]), std=tensor([0.5000, 0.5000, 0.5000]))
)
```
=> here, [ToTensor](https://pytorch.org/vision/stable/generated/torchvision.transforms.ToTensor.html) is used, which as stated in the docs converts a tensor with values 0 - 255 to the range [0-1]. Maybe @rwightman can clarify this.

Regarding point 2:

That's a valid point, not sure why I missed that. We could update this, although I'm not sure it has a big impact on downstream results. cc @sgugger 

@NielsRogge @zhoutong-fu

ToTensor: uint8 [0, 255] -> float [0, 1.0]
Normalize(mean=tensor([0.5000, 0.5000, 0.5000]), std=tensor([0.5000, 0.5000, 0.5000])): float [0, 1.0] -> float [-1, 1]

Interpolation should be bicubic for both ViT and ViT MAE




Thank you guys for the discussion.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.