noamaanMulla-03/transformers-issues huggingface dataset:
Decoding throws Segmentation Fault 
Author: macabdul9
Pull request: no

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 4.4.2
- Platform: Linux-5.8.0-48-generic-x86_64-with-glibc2.10
- Python version: 3.8.5
- PyTorch version (GPU?): 1.8.1 (True)
- Tensorflow version (GPU?): 2.4.1 (False)
- Using GPU in script?: <fill in>
- Using distributed or parallel set-up in script?: <fill in>


Models:

- bert


## Information

A very simple decoding step throws `24272 segmentation fault (core dumped)` .

## To reproduce

Steps to reproduce the behavior:

```
from transformers import AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained("bert-base-uncased")
print(tokenizer.decode(token_ids=torch.tensor([3446])))
```

<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

## Expected behavior

It should print `'rate'`


Comments:
I get `rate` printed out on my setup! Do you mind sharing your `pip list`?

This was mentioned in #4857 and #5359 ....fiixed it after reinstalling. 

For me, i didn't have sentencepiece installed. I needed to import torch before transformers to fix this
Fix `Mask2FormerForUniversalSegmentation` and failed tests
Author: ydshieh
Pull request: yes

# What does this PR do?

For `Mask2FormerForUniversalSegmentation` , the tests `test_torchscript_xxx` fails due to `auxiliary_logits` in `Mask2FormerForUniversalSegmentationOutput`. This is a `list of dict of tensors`, which is not supported by torchscript tracing.

The related tests will pass if we don't output this value.  Currently, we have

```python
        output_auxiliary_logits = (
            self.config.use_auxiliary_loss if output_auxiliary_logits is None else output_auxiliary_logits
        )
```
where `use_auxiliary_loss` is `True`. However,  this seems strange to me, and I believe it should be `self.config.output_auxiliary_logits` instead. So I change it.

And this also fixes the related tests too, as `output_auxiliary_logits` is `None` (during the tests)


Comments:
_The documentation is not available anymore as the PR was closed or merged._
Fix AdamWeightDecay for TF 2.11
Author: Rocketknight1
Pull request: yes

Fixes #20724 

Comments:
@gante take a quick look at this one too, please! Our `AdamWeightDecay` doesn't work with the new Optimizer base class in TF 2.11, so I moved it onto the legacy class for now. When I have more time I should clean this up properly, and possibly just use [the official AdamW](https://www.tensorflow.org/api_docs/python/tf/keras/optimizers/experimental/AdamW) instead.

_The documentation is not available anymore as the PR was closed or merged._

The same issue occurred with the Summarization tutorial. [https://huggingface.co/course/chapter7/5?fw=tf#models-for-text-summarization](url)

Thanks @el-profesor-04 - can you confirm that the issue goes away with you install the latest version of `transformers` from main with `pip install --upgrade git+https://github.com/huggingface/transformers.git`?
Rename GLPN image processor tests
Author: amyeroberts
Pull request: yes

# What does this PR do?

Renames GLPN feature extractor tests. Missed file in #21140 

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


Comments:
_The documentation is not available anymore as the PR was closed or merged._
[Whisper] Fix timestamp processor
Author: ArthurZucker
Pull request: yes


# What does this PR do?
Mostly adds condition when looking in the past and in the futur based on timing information. 
Fixes the tests

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_21187). All of your documentation changes will be reflected on that endpoint.

Tested with a concatenated librispeech (clean, test, 5.4 hours), took 393.1348168849945 seconds, and with a WER 0.030776774096215136. So in that case not really sure why we are performing better.
Openai took 2661.047640323639  and had a wer of 0.2589624153733004. Which is pretty interesting (the model is `large`) so a `x6.7`speed



Will open a PR for the other fix
OPT: Fix batched generation with FLAX
Author: gante
Pull request: yes

# What does this PR do?

Fixes #20666 

OPT numerical masking was using `-inf` where the attention mask was `0`, which in turn caused the hidden states to be `nan` and derail the whole generation. Changing to a common masking value (`-1e9`) fixes the issue. I've also taken the opportunity to re-enable the commented out tests :)

Comments:
_The documentation is not available anymore as the PR was closed or merged._
Flax dtype-dependent numerical masking
Author: gante
Pull request: yes

# What does this PR do?

Fixes #21176

For some models, our Flax numerical masking was incompatible with the desired variable type. This PR fixes it by selecting a numerical mask that is the minimum for the corresponding variable type.

This PR is akin to #17306 for PT. Thank you @LysandreJik and @ydshieh for pointing it out 🙏 

Comments:
@sgugger this solution was discussed on Slack with the Flax team, hence no added Flax reviewers :)

_The documentation is not available anymore as the PR was closed or merged._
Add disclaimer for necessary fake models
Author: sgugger
Pull request: yes

# What does this PR do?

This PR adds a disclaimer for the fake models we can't really remove since the canonical checkpoint is very big already. You can see the result in the [preview](https://moon-ci-docs.huggingface.co/docs/transformers/pr_21178/en/model_doc/gptj#transformers.GPTJModel.forward.example).

If this is acceptable, I'll add it to the other places where we want a tiny random model instead of the huge one for the docstrings.

Comments:
_The documentation is not available anymore as the PR was closed or merged._
EfficientFormer
Author: jhabr
Pull request: no
Labels: name: New model

### Model description

[EfficientFormer: Vision Transformers at MobileNet Speed](https://arxiv.org/abs/2206.01191)

Submitted on 2 Jun 2022, last revised 5 Jul 2022.

### Open source status

- [X] The model implementation is available
- [X] The model weights are available

### Provide useful links for the implementation

https://github.com/snap-research/EfficientFormer

Happy to support the model contribution!

Comments:
cc @hollance
Update modeling doc strings FE -> IP
Author: amyeroberts
Pull request: yes

# What does this PR do?

Replaces references of feature extractors with image processors that had been missed in the first batch of changes 

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


Comments:
_The documentation is not available anymore as the PR was closed or merged._
Typo in XCLIP model
Author: Zhilin123
Pull request: no

### System Info

transformers version 4.25.1



### Who can help?

@NielsRogge @ydshieh 

There's a typo/mismatch for the xclip model pretrained config archive map between https://github.com/huggingface/transformers/blob/main/src/transformers/models/auto/configuration_auto.py#L333

and. https://github.com/huggingface/transformers/blob/main/src/transformers/models/x_clip/configuration_x_clip.py#L27

notice in the first example, there is underscore between X and CLIP whereas in the second example, there isn't.

This leads to an error when initializing automodel


`   
    huggingface_models = list(ALL_PRETRAINED_CONFIG_ARCHIVE_MAP.keys())

  File "/usr/local/lib/python3.8/dist-packages/transformers/models/auto/configuration_auto.py", line 612, in keys

    self._initialize()

  File "/usr/local/lib/python3.8/dist-packages/transformers/models/auto/configuration_auto.py", line 602, in _initialize

    mapping = getattr(module, map_name)

  File "/usr/local/lib/python3.8/dist-packages/transformers/utils/import_utils.py", line 1086, in __getattr__

    raise AttributeError(f"module {self.__name__} has no attribute {name}")

AttributeError: module transformers.models.x_clip has no attribute X_CLIP_PRETRAINED_CONFIG_ARCHIVE_MAP`

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

1. from transformers import ALL_PRETRAINED_CONFIG_ARCHIVE_MAP
2. huggingface_models = list(ALL_PRETRAINED_CONFIG_ARCHIVE_MAP.keys())

### Expected behavior

Expect no errors to be raised

Comments:
@Zhilin123 Thank you for reporting. See #21207. However, note that 

```
ALL_PRETRAINED_CONFIG_ARCHIVE_MAP is deprecated and will be removed in v5 of Transformers.
```
Fix `CONFIG_ARCHIVE_MAP_MAPPING_NAMES`
Author: ydshieh
Pull request: yes

# What does this PR do?

Fix `CONFIG_ARCHIVE_MAP_MAPPING_NAMES` as reported in #21204.
Also, `UPERNET_PRETRAINED_CONFIG_ARCHIVE_MAP` doesn't exist.

Remark: we have planned deprecation
```
        warnings.warn(
            "ALL_PRETRAINED_CONFIG_ARCHIVE_MAP is deprecated and will be removed in v5 of Transformers. "
            "It does not contain all available model checkpoints, far from it. Checkout hf.co/models for that.",
            FutureWarning,
        )
```

Fix #21204

Comments:
_The documentation is not available anymore as the PR was closed or merged._

I will merge as it is - don't think it matters (at least not at this moment), and eventually this is going to be deprecated.
Fix GPTJ doctest
Author: ydshieh
Pull request: yes

# What does this PR do?

In #21178, the checkpoint used for `GPTJForSequenceClassification` is changed back to `_CHECKPOINT_FOR_DOC`, which is `hf-internal-testing/tiny-random-gptj`. That checkpoint has `self.score` with shape `[2, 512]`, but the model has `self.score` with shape `[2, 32]` as `config.n_embd=32`. The checkpoint has `512` which came from a mistake of using `n_ctx` previously, and that error is fixed in #14190, see [here](https://github.com/huggingface/transformers/commit/ce91bf9a3431b4d260005de84c0b0fa394409a3c#diff-61155574bf9c9669ccdfdf7dd508a5979b4e4915cc95f7ff4a63fee05a0e2715).

The PR uses another tiny checkpoint to pass the test.



Comments:
_The documentation is not available anymore as the PR was closed or merged._
Declare __len__ method in PreTrainedTokenizerBase
Author: thomasw21
Pull request: yes

# What does this PR do?

When type hinting a tokenizer with `PreTrainedTokenizerBase`, it doesn't know supprt `len(tokenizer)`, but both slow and fast version implement `__len__` so we declare it at least to make the type hints happy, we could make this an `abstract_method` as well if needed

Comments:
_The documentation is not available anymore as the PR was closed or merged._
Fix code example in training tutorial
Author: stevhliu
Pull request: yes

The Keras [section](https://huggingface.co/docs/transformers/main/en/training#train-a-tensorflow-model-with-keras) of the training tutorial throws an error because it tokenizes `dataset["text"]` instead of `dataset["sentence"]`. There is no `text` column in this dataset.

Comments:
_The documentation is not available anymore as the PR was closed or merged._
Make `parallelism` for CircleCI jobs work - but keep it to be `1` for now
Author: ydshieh
Pull request: yes

# What does this PR do?

Enable `parallelism` for CircleCI jobs. So far I only enable it for torch/tf/flax jobs. It can be switch off easily.

Comments:
_The documentation is not available anymore as the PR was closed or merged._

> Thanks for looking into this. As said internally I'm really not a fan of the split test reports this gets us at the end.

I can probably try to concatenate the failed tests (with details) at the end - although so far I don't have clear idea of the feasibility)

But at least we can switch back to N=1 if we really need to (i.e. we have real difficulty to read the test failures)

Ready for review again. So far it still uses `parallelism=1` - this PR just provides the necessary change for using `parallelism=N`. Hopefully I can figure out a way to make a better report format in another PR, and we can finally go for N > 1.

> Thanks! Another thing to look for before enabling it is if we just pay 8x the price for parallelism=8 and not way more than that.

@sgugger I think the expectation to check is if we just pay 1x~2x the price for parallelism=N (say 8). Using 8 executors means less runtime on each executor (ideally 1/8), so the total should be the same (but there is definitely some overhead, like in cache loading / pip install steps)
Fix OneFormer Docstrings
Author: praeclarumjj3
Pull request: yes

# What does this PR do?

Fixes docstrings for OneFormer.

- [x] Checked that all the doctests passed for the following command:

```bash
python3 -m pytest -v --make-reports doc_tests_gpu --doctest-modules src/transformers/models/oneformer/ -sv --doctest-continue-on-failure --doctest-glob="*.mdx"
```
## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

@ydshieh 

Comments:
_The documentation is not available anymore as the PR was closed or merged._
Fix task summary doctest 
Author: stevhliu
Pull request: yes

The doctests are failing for the updated task summary page because outputs weren't included in the code examples. This PR adds real inputs that can be pipelined instead of the generic `("path/to/data/file")`. I skipped the `text-generation` pipeline, but let me know if you'd prefer setting a seed for it so we can still reliably generate an output.

Comments:
_The documentation is not available anymore as the PR was closed or merged._
rename configuration_utils.PretrainedConfig.max_length to max_generation_length
Author: aaronrmm
Pull request: no

### Feature request

Max length (and min length) is an overloaded term. This renaming would help disambiguate this parameter from tokenizer max lengths when parameters are logged.



https://github.com/huggingface/transformers/blob/862888a35834527fed61beaf42373423ffdbd216/src/transformers/configuration_utils.py#L119

### Motivation

When looking at Mlflow logs for my text classification model, I saw "max length" as one of the parameters. I had to debug to figure out that it was not relevant to classification. This seems to me like the simplest solution to make it clear that it is a parameter that is only relevant to generation.

### Your contribution

I can of course do a PR with the renaming change. However, I am not sure of the downstream implications.

Comments:
Thanks for raising an issue. Renaming an argument like this which is so widely used is too breaking a change for us to consider however. 

Makes sense. Perhaps there is something that can be done in the mlflow logger instead. I'll raise a different issue if I have any ideas.
[`BLIP`] fix docstring for `BlipTextxxx`
Author: younesbelkada
Pull request: yes

# What does this PR do?

Fixes docstrings for `BlipTextModel` and `BlipTextLMHeadModel` to follow the dostring structure of `transformers` and be rendered properly by the `doc-builder`

cc @sgugger 

Comments:
_The documentation is not available anymore as the PR was closed or merged._
Skip failing test for now
Author: sgugger
Pull request: yes

# What does this PR do?

All is said in the title. Test is currently failing on main for no reason (I imagine a new release of one of the deps), more can be found [here](https://app.circleci.com/pipelines/github/huggingface/transformers/55784/workflows/83b929a9-3d0d-482a-a823-806f44824bf8/jobs/673138).

cc @sanchit-gandhi 

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_21226). All of your documentation changes will be reflected on that endpoint.
MobileViT
Author: ludmila3
Pull request: no

### System Info

- `transformers` version: 4.26.0.dev0
- Platform: Linux-5.15.0-1027-gcp-x86_64-with-debian-bullseye-sid
- Python version: 3.7.12
- Huggingface_hub version: 0.11.0
- PyTorch version (GPU?): 1.12.1 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: <fill in>
- Using distributed or parallel set-up in script?: <fill in>

### Who can help?

 @amyeroberts and @NielsRogge

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

1.Run run_image_classification.py with MobileViT models (mobilevit-x-small)


### Expected behavior

Breaks first on Normalize function


Comments:
Answered here: https://github.com/NielsRogge/Transformers-Tutorials/issues/241
The wrong expression in function "squad_convert_example_to_features"
Author: chen-huanxin
Pull request: no

https://github.com/huggingface/transformers/blob/d4bf9ee1ff0e85cb24feec4dd160af39b623d4b9/src/transformers/data/processors/squad.py#L253
The type of `span["input_ids"]` is `List`, and the `tokenizer.pad_token_id` equals 0, so `span["input_ids"] == tokenizer.pad_token_id` returns bool type result: `False`. It leads to `pad_token_indices = np.where(False)` to be empty array. This may not be the ideal outcome. Because the latter expression 
https://github.com/huggingface/transformers/blob/d4bf9ee1ff0e85cb24feec4dd160af39b623d4b9/src/transformers/data/processors/squad.py#L258
 uses the `pad_token_indices` to generate p_mask, which should be a `np.ndarray` that sets the padding item in it to 1.
I think the right expression may be as follow:
```Python
pad_token_indices = np.where(np.array(span["input_ids"]) == tokenizer.pad_token_id)
```
`np.array(span["input_ids"])` make a `np.ndarray`, which can compare with `tokenizer.pad_token_id`, and generate `np.ndarray` result. It make sence of expression `p_mask[pad_token_indices] = 1`. 
My English is poor, I hope you can understand my words. Thanks!

Comments:
This is all legacy code that we are not maintaining anymore though.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
"Loading weights from local directory" 
Author: ChingKwanCheung
Pull request: no

I've been stuck in "Loading weights from local directory" when run the code(https://github.com/huggingface/transformers/blob/main/examples/research_projects/mlm_wwm/run_chinese_ref.py). My cmd for running run_chinese_ref.py is below:
![image](https://user-images.githubusercontent.com/30768203/207237120-4059d721-e8cb-4291-9e24-cc26a60af918.png)
![image](https://user-images.githubusercontent.com/30768203/207396256-6c3e0a2f-5207-4617-a6b9-f583b8863c19.png)


Comments:
I download the LTP model from https://huggingface.co/LTP/small/tree/main . And put it in the LTP-small local folder.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Transformer cannot tokenize Chinese words
Author: fivehills
Pull request: no


Comments:
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

There is nothing we can do without a code reproducer and, in particular, knowing which model you're using.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Statement seems to have no effect
Author: daskol
Pull request: no

### System Info

transformers from `v4.3.3` to `v4.25.1`.

### Who can help?

@sgugger

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

Just stare at [this line][1] (method is not actually called).
```python
class Trainer:
    def __init__(self, ...)
        ...

        # force device and distributed setup init explicitly
        args._setup_devices

        ...
```
This change was done in 2021-02-11 (almost two years ago).

[1]: https://github.com/huggingface/transformers/blob/4e730b387364c9f46b6b1b0c79fdaf0903c42257/src/transformers/trainer.py#L329

### Expected behavior

I do not know what to expect because of the issue. May be all distributed (at least parallel) training with PyTorch is broken. May be everything is fine. I am totally not sure. I'd like to see some regression tests or something that prove that there is not issue or something what was broken after this change.

Comments:
It turns out that this is not a function but a property which do some complex initialization. :shrug: