noamaanMulla-03/transformers-issues huggingface dataset:
[WIP] Fix `assert` statements found in modeling files
Author: cwkeam
Pull request: yes

# What does this PR do?

Fixing unnecessary `assert` statements under modeling code. This was motivated by the fact that contributing to `transformers` often requires a lot of copying from existing code, and while `assert` statements are [undesirable in new PRs](https://github.com/huggingface/transformers/pull/16402#discussion_r881715669), a lot of existing code includes `assert` statements.

I was just going to fix it for `Speech2Text` as referenced above, but found many instances of it throughout the codebase.

It's actually not entirely clear to me where `assert` is not desirable. For utility functions like `convert_*_utils.py` it seems forgivable (used by advanced developers anyway), but correct me if I'm wrong (otherwise there are hundreds of instances to fix).  

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

@sgugger 


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_17598). All of your documentation changes will be reflected on that endpoint.

Can you solve the merge conflicts? This should re-trigger the CI and we will see if the errors persist (there were some outage on the Hub this weekend).

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Fix beam search computing wrong `next_indices`
Author: m43
Pull request: yes

# What does this PR do?

Beam search used `next_indices = (next_tokens / vocab_size).long()` to compute the indices of the best beams. This, however, uses [torch.true_divide](https://pytorch.org/docs/stable/generated/torch.true_divide.html#torch-true-divide) which would lead to numerical errors and make the following snippet fail:
```py
import torch

next_tokens = torch.tensor([[0, 50257]], dtype=torch.int64, device='cuda:0')
vocab_size = 50257
expected_next_indices = torch.tensor([[0,1]], dtype=torch.int64, device='cuda:0')

next_indices = (next_tokens / vocab_size).long()
print(next_indices)

assert torch.all(next_indices == expected_next_indices)  # Fails
```

The simple fix in this PR uses floor division to avoid the aforementioned problem:
```py
# ...
next_indices = torch.div(next_tokens, vocab_size, rounding_mode='floor').long()
print(next_indices)

assert torch.all(next_indices == expected_next_indices)  # Passes
```

I have dug out this bug while recomputing the beam search scores by hand for gpt2. If needed, I can add an end-to-end high-level reproducibility test with gpt2 and the tricky input_ids, and possibly add it as a unit test.

## Who can review?

@patrickvonplaten

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_18165). All of your documentation changes will be reflected on that endpoint.
How to override model.generate() function in GenerationMixin class?
Author: yeounyi
Pull request: no

Hi, I want to override `_generate_no_beam_search`, `_generate_beam_search` methods in `GenerationMixin` class to adjust `next_token_logits`. 
I tried it by adding adjusted methods in my custom model code but seems not working. 
I'd appreciate any help. Thanks! 

Comments:
Hello, thanks for opening an issue! We try to keep the github issues for bugs/feature requests.
Could you ask your question on the [forum](https://discuss.huggingface.co) instead?

Thanks!

Hi, have you found the solutions? I'm facing the same issue.

@rxlian Hi, I failed to override `model.generate()` function, but in my case overriding `_prepare_encoder_decoder_kwargs_for_generation` function in the model definition was successful!
Enable Past CI
Author: ydshieh
Pull request: yes

# What does this PR do?

Enable Past CI

Comments:
_The documentation is not available anymore as the PR was closed or merged._

> merge it ... and monitor failures. 

I read your comment too quickly. So far the Past CI will be triggered only on pushing to `run-past-ci*` branches. I ran it ~ June 20 however, and I opened #18181 today.

I think we can launch past CI monthly or even bimonthly. Please let me know if you have different opinion, @LysandreJik. Thanks.
the Sigopt api is outdated in transformers trainer.py, the old api could not work
Author: sywangyi
Pull request: no
Labels: name: bug

### System Info

- `transformers` version: 4.21.0.dev0
- Platform: Linux-5.8.0-43-generic-x86_64-with-glibc2.29
- Python version: 3.8.10
- Huggingface_hub version: 0.7.0
- PyTorch version (GPU?): 1.11.0+cu113 (True)
- Tensorflow version (GPU?): 2.9.1 (False)
- Flax version (CPU?/GPU?/TPU?): 0.5.0 (cpu)
- Jax version: 0.3.6
- JaxLib version: 0.3.5
- Using GPU in script?: <fill in>
- Using distributed or parallel set-up in script?: <fill in>


### Who can help?

@sgugger 

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

1.enable sigopt HPO in example and run.
2. work log like"UserWarning: You're currently using the old SigOpt Experience. Try out the new and improved SigOpt experience by getting started with the docs today. You have until July 2022 to migrate over without experiencing breaking changes."

### Expected behavior

HPO with sigopt backend could work correctly without warning

Comments:
add ONNX support for LeViT
Author: gcheron
Pull request: yes

# What does this PR do?

This PR adds ONNX support for LeViT. Linked to [#16308](https://github.com/huggingface/transformers/issues/16308).

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [x] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [x] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- albert, bert, xlm: @LysandreJik
- blenderbot, bart, marian, pegasus, encoderdecoder,  t5: @patrickvonplaten, @patil-suraj
- longformer, reformer, transfoxl, xlnet: @patrickvonplaten
- fsmt: @stas00
- funnel: @sgugger
- gpt2: @patrickvonplaten, @LysandreJik
- rag: @patrickvonplaten, @lhoestq
- tensorflow: @LysandreJik

Library:

- benchmarks: @patrickvonplaten
- deepspeed: @stas00
- ray/raytune: @richardliaw, @amogkam
- text generation: @patrickvonplaten
- tokenizers: @n1t0, @LysandreJik
- trainer: @sgugger
- pipelines: @LysandreJik

Documentation: @sgugger

HF projects:

- datasets: [different repo](https://github.com/huggingface/datasets)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Examples:

- maintained examples (not research project or legacy): @sgugger, @patil-suraj
- research_projects/bert-loses-patience: @JetRunner
- research_projects/distillation: @VictorSanh

 -->


Comments:
_The documentation is not available anymore as the PR was closed or merged._

Pinging @lewtun and @sgugger for approval.
One CI test failed but it doesn't seem to be related to this PR.

You are welcome, thank you for your work!
Yes, following the doc I have already run this command and it passed all (slow) tests ;)
Fix expected loss values in some (m)T5 tests
Author: ydshieh
Pull request: yes

# What does this PR do?

Fix CI failures regarding some T5 and MT5 tests.

The PR #18013 and the subsequent fix in #18029  probably tried to get the expected loss values without setting

```python
os.environ["NVIDIA_TF32_OVERRIDE"] = "0"
```

Comments:
_The documentation is not available anymore as the PR was closed or merged._
Update TF(Vision)EncoderDecoderModel PT/TF equivalence tests
Author: ydshieh
Pull request: yes

# What does this PR do?

Make PT/TF equivalence tests for TF(Vision)EncoderDecoderModel aligned with the ones defined in `test_modeling_tf_common.py`:
- test all hidden states, attention outputs
- (and a minor bonus: consistent test names)

Comments:
_The documentation is not available anymore as the PR was closed or merged._

> Would it be possible to move in the opposite direction, i.e. make the TFEncoderDecoder tests (or TFEncoderDecoderMixin itself) inherit from TFModelTesterMixin?

I also asked myself, and think it would be nice. Having something like `TFGenericEncoderDecoderModelTesterMixin` (or whatever  better names) and inheriting from there is good enough (see below). Regarding inheriting from `TFModelTesterMixin`, we have to skip some (or a lot of) tests that are not for encoder-decoder architectures, and requires some extra changes in `TFModelTesterMixin` to take into account encoder-decoder architectures. 

There is a previous related discussion:

- one from @patrickvonplaten (and more related to this discussion here): https://github.com/huggingface/transformers/pull/16280#issuecomment-1077449865
- from @sgugger https://github.com/huggingface/transformers/pull/16280#issuecomment-1077603150
- a thread: https://github.com/huggingface/transformers/pull/16280#discussion_r832036543

remark: the discussion regarding `is_encoder_decoder` there is (originally) more about Bart / T5 etc, not the `(Vision/Speech)EncoderDecoderModel`.

Okay I see the argument and I agree with it -- several composable mixins would be the way!

We may be able to carve out some tests from `TFModelTesterMixin` that are truly global, but streamlining `EncoderDecoder` tests seems more important 👍 I will be reviewing this PR soon with that in mind

@gante I also fixed `closed enough` in `tests/test_modeling_common.py` (thinking it is tiny that could be fixed in this PR too)
Better default for offload_state_dict in from_pretrained
Author: sgugger
Pull request: yes

# What does this PR do?

Seeing issues arise since the release of big model inference, I realized it's very confusing for users to have to set `offload_state_dict=True` when the device map picked with `device_map="auto"` contains some disk-offloaded weights. Therefore, this PR changes the default to `None` to pick a good default (basically `False` if there is no disk offload and `True` otherwise) while still letting the user choose the behavior they want by passing a value.

Comments:
_The documentation is not available anymore as the PR was closed or merged._
Can we do adaptive pretraining on BERT-related models using transformers?
Author: dr-GitHub-account
Pull request: no

### Feature request

Adaptive pretraining methods like domain adaptive pretraining and task adaptive pretraining can benefit downstream tasks, which is illustrated in [https://aclanthology.org/2020.acl-main.740.pdf](url). In [https://huggingface.co/models](url), there are successful models pretrained with data from source domain. I would like to do adaptive pretraining(with tasks like MLM) to chinese-roberta-wwm-ext-large: [https://huggingface.co/hfl/chinese-roberta-wwm-ext-large](url) using unlabeled target domain data, so as to get better result in downstream tasks.

### Motivation

BERT and related models are benefiting some areas. The following are some examples:
1. [http://arxiv.org/abs/2004.02288](url)
2. [http://arxiv.org/abs/1908.10063](url)
3. [http://arxiv.org/abs/2007.15779](url)
4. [http://arxiv.org/abs/1906.02124](url)
5. [http://arxiv.org/abs/1904.05342](url)
I would like to do adaptive pretraining to chinese-roberta-wwm-ext-large, using unlabeled Chinese data in my area. It is good to start with conventional MLM. Afterwards, I might try and follow the pretraining task setting of chinese-roberta-wwm-ext-large, i.e. whole word masking and dynamic masking.

### Your contribution

Hopefully, a domain-specific pretrained language model.

Comments:
Hello, thanks for opening an issue! We try to keep the github issues for bugs/feature requests.
Could you ask your question on the [forum](https://discuss.huggingface.co) instead?

Thanks!

I will. Thanks for the instruction.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Rm hardcoded style fromdoc expert acc imgs
Author: mishig25
Pull request: yes

Having a hardcoded style of `width:600px` is overriding hf.co doc styles for images, which is causing an overflowing problem on mobile

| before | after |
|--------|-------|
|    <img width="400" alt="Screenshot 2022-06-10 at 09 53 20" src="https://user-images.githubusercontent.com/11827707/173018283-a1d5e103-4bd6-4814-af25-f0d0fc9f3888.png">    |    <img width="400" alt="Screenshot 2022-06-10 at 09 53 12" src="https://user-images.githubusercontent.com/11827707/173018274-2227571a-702a-4c82-b103-3a5a1a4ae029.png">   |

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_17648). All of your documentation changes will be reflected on that endpoint.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Ensure image feature extractors outputs pixel values of shape (batch_size, n_channels, height, width)
Author: alaradirik
Pull request: yes
Labels: name: Feature extractors

Fixes #15055 #17526 

- center_crop in image_utils.py now can process numpy and torch images of shape (n_channels, height, width) and (height, width, n_channels), and returns cropped image of shape (n_channels, height, width).
- Fixed feature extractors that inherit from ImageFeatureExtractionMixin to always return numpy arrays or tensors of shape (batch_size, n_channels, height, width)

Comments:
Hi, @alaradirik. Thank you for the (first) PR! I have a few suggestions here :-)

1. It would be great if we could have a more descriptive PR title.
2. Each PR should focus on a single issue.
    - **Unless** the issues are tightly coupled, and the changed really have to be done simultaneously otherwise things break.
    - (I didn't go through the changes in this PR though) 
3. If 2.) could be achieved, it's more likely easier to find a descriptive titles for each PR.

I don't mean that we should close this PR and open new ones, I will let the reviews to decide.
But if you are able to think of a good PR title, it would be very nice. Thank you.

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_17620). All of your documentation changes will be reflected on that endpoint.

> Hi, @alaradirik. Thank you for the (first) PR! I have a few suggestions here :-)
> 
> 1. It would be great if we could have a more descriptive PR title.
> 2. Each PR should focus on a single issue.
>    
>    * **Unless** the issues are tightly coupled, and the changed really have to be done simultaneously otherwise things break.
>    * (I didn't go through the changes in this PR though)
> 3. If 2.) could be achieved, it's more likely easier to find a descriptive titles for each PR.
> 
> I don't mean that we should close this PR and open new ones, I will let the reviews to decide. But if you are able to think of a good PR title, it would be very nice. Thank you.


Thank you! I will keep these in mind in my next PR.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
model.parallelize() for OPT models 
Author: MikeWangWZHL
Pull request: no

### Feature request

It would be great if we can have function for fitting OPT model on multiple GPUs using **model.parallelize()**, similar to what we already have for [GPT-J](https://github.com/huggingface/transformers/blob/6e535425feae20ca61a8b10ae5e8a7fab4d394ba/src/transformers/models/gptj/modeling_gptj.py#L494)

### Motivation

It would be extremely helpful for fitting large OPT models such as opt-30b, opt-13b. An ideal solution would be similar to what has been integrated in [GPT-J](https://github.com/huggingface/transformers/blob/6e535425feae20ca61a8b10ae5e8a7fab4d394ba/src/transformers/models/gptj/modeling_gptj.py#L494), where we can simply call **model.parallelize()** to load the model on multiple gpus.

### Your contribution

glad to help on making this feature possible

Comments:
@sgugger can share more on that, but we now recommend leveraging `accelerate` in order to parallelize models. See [tweet](https://twitter.com/huggingface/status/1524783489593360385) that contains a [colab example](https://colab.research.google.com/drive/14wnxMvD9zsiBQo2FtTpxn6w2cpXCcb-7#scrollTo=XUlpcU3iQNhu&uniqifier=1).

On Transformers main branch, you can also load the OPT models on several GPUs directly with `AutoModel.from_pretrained(checkpoint, device_map="auto")` (or pass along you own device map).

The old `parallelize` API will be deprecated soon.

@sgugger Hi, that's very helpful for me. However I wonder what's the difference between directly using ```AutoModel.from_pretrained(checkpoint, device_map="auto")```  and that using ```load_checkpoint_and_dispatch()``` and ```infer_auto_device_map()```? Does both of them support fine-tuning something like OPT-30B on multiple GPUs with model parallelism? Thanks a lot for your help!

No this is inference only. For training/fine-tuning we recommend the use of DeepSpeed Zero-3.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
ImportError: cannot import name 'BloomTokenizer' from 'transformers'
Author: rhvaz
Pull request: no
Labels: name: bug

### System Info

transformers==4.20.1
torch==1.12.0
Python 3.9.13
GPU: yes (running on GCP)

### Who can help?

@SaulLu 

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

```
from transformers import BloomTokenizer, BloomModel
```

I was following https://huggingface.co/docs/transformers/model_doc/bloom

Specifically 
```
from transformers import BloomTokenizer, BloomModel
import torch

tokenizer = BloomTokenizer.from_pretrained("bigscience/Bloom")
model = BloomModel.from_pretrained("bigscience/Bloom")

inputs = tokenizer("Hello, my dog is cute", return_tensors="pt")
outputs = model(**inputs)

last_hidden_states = outputs.last_hidden_state
```

### Expected behavior

No import error.

Comments:
Hi @rhvaz,

I am sincerely sorry that you have encountered this issue. We do have a small typo in our documentation, which has been resolved in the PR https://github.com/huggingface/transformers/pull/18005 but not yet deployed on our website.

In the meantime, here is the snippet that should work:
```python
from transformers import BloomTokenizerFast, BloomModel
import torch

tokenizer = BloomTokenizerFast.from_pretrained("bigscience/Bloom")
model = BloomModel.from_pretrained("bigscience/Bloom")

inputs = tokenizer("Hello, my dog is cute", return_tensors="pt")
outputs = model(**inputs)

last_hidden_states = outputs.last_hidden_state
```
 

Hi @SaulLu many thanks for looking at this so quickly!

I tried the snippet you shared and I am now having the following permission issues
```
OSError: bigscience/Bloom is not a local folder and is not a valid model identifier listed on 'https://huggingface.co/models'
If this is a private repository, make sure to pass a token having permission to this repo with `use_auth_token` or log in with `huggingface-cli login` and pass `use_auth_token=True`.
```

I get the exception when I try to run either of the lines below
```
tokenizer = BloomTokenizerFast.from_pretrained("bigscience/Bloom")
model = BloomModel.from_pretrained("bigscience/Bloom")
```

You spotted another typo in the name of the checkpoint! Here's a new snippet that should work:
```python
from transformers import BloomTokenizerFast, BloomModel
import torch

tokenizer = BloomTokenizerFast.from_pretrained("bigscience/bloom")
model = BloomModel.from_pretrained("bigscience/bloom")

inputs = tokenizer("Hello, my dog is cute", return_tensors="pt")
outputs = model(**inputs)

last_hidden_states = outputs.last_hidden_state
```

I also took this opportunity to share the same fix in the documentation in PR #18187