noamaanMulla-03/transformers-issues huggingface dataset:
Pytorch 2 generation/utils.py , 'torch.distributed' has no attribute 'world_size'
Author: djaym7
Pull request: no

### System Info
transformers 4.28.0.dev0
pytorch 2 
cuda 117

File "/usr/local/lib/python3.8/dist-packages/transformers/generation/utils.py", line 1196, in generate
    if is_deepspeed_zero3_enabled() and dist.world_size() > 1:
AttributeError: module 'torch.distributed' has no attribute 'world_size'



https://github.com/huggingface/transformers-bloom-inference/blob/7bea3526d8270b4aeeefecc57d7d7d638e2bbe0e/bloom-inference-scripts/bloom-ds-zero-inference.py

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

Pytorch 2 and generate with deepspeed stage 3. 
https://github.com/huggingface/transformers-bloom-inference/blob/7bea3526d8270b4aeeefecc57d7d7d638e2bbe0e/bloom-inference-scripts/bloom-ds-zero-inference.py

### Expected behavior

No error 

Comments:
Looks like get_world_size() is supposed to be used in pytorch.distributed. [I will make a PR.](https://github.com/huggingface/transformers/pull/22381)

Should be fixed now that the PR above has been merged.
[`bnb`] Force `requires_grad` to be `False`
Author: younesbelkada
Pull request: yes

# What does this PR do?

Addresses: https://github.com/huggingface/accelerate/pull/1237#discussion_r1146045485

Some users uses `replace_8bit_linear` as a standalone function, by default the `Linear8bitLinear` layers that are newly created have `requires_grad` set to `True` leading to a bug on `bitsandbytes` library. 

This PR forces `requires_grad` to be `False` for these layers to avoid these issues 

Related: https://github.com/huggingface/accelerate/pull/1237

All bnb slow tests pass

cc @sgugger 

Comments:
_The documentation is not available anymore as the PR was closed or merged._
Add BioGptForSequenceClassification
Author: GuillemGSubies
Pull request: yes

# What does this PR do?

Fixes #21530

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [x] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [x] Did you write any new necessary tests?


## Who can review?

@NielsRogge


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_21535). All of your documentation changes will be reflected on that endpoint.

Hi, it seems the CI didn't run properly. Could you push an empty commit to trigger it?

What Niels forgot to say is that there seems to be an issue with your CircleCI permissions, the tests won't run.
Could you try refreshing your permissions as shown [here](https://support.circleci.com/hc/en-us/articles/360048210711-How-to-Refresh-User-Permissions-) and then push some commit to your PR?

> What Niels forgot to say is that there seems to be an issue with your CircleCI permissions, the tests won't run. Could you try refreshing your permissions as shown [here](https://support.circleci.com/hc/en-us/articles/360048210711-How-to-Refresh-User-Permissions-) and then push some commit to your PR?

I tried but it didn't work 😖 


Hi @GuillemGSubies. If the issue is still open, i would like to contribute it.
Thanks

@upjabir I don't understand why the pipelines fail, you have any idea? Otherwise you can open another PR and I'll close this one

@GuillemGSubies currently I don't have any idea about the issue. But let me try it, so I can get a clear picture

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Report safetensors version in transformers-cli env
Author: sgugger
Pull request: yes

# What does this PR do?

This PR adds `safetensors` to the info reported by `transformers-cli env` and in particular puts a note when safetensors is ignored becuase of a too old PyTorch (see #22370 )

Comments:
_The documentation is not available anymore as the PR was closed or merged._
Hardware Auto-Setup for Examples
Author: dongreenberg
Pull request: yes

# What does this PR do?

Discussed with @sgugger and @LysandreJik . This PR introduces auto-setup functionality for tutorials and examples (we are sending a parallel PR to accelerate, and maybe Diffusers and Spaces shortly). This allows users to run transformers code, tutorials, and scripts on self-hosted hardware (either their own instance or a cloud instances) including on-demand allocation of the hardware itself on AWS, GCP, Azure, or Lambda Labs, and installation of dependencies. This introduces a similar level of turnkey usage and reproducibility that users only typically expect in Colab, but for any type of hardware on any cloud (we've tested on Paperspace and Coreweave as well, allocating the instance in their UI and then plugging in the IP as a static cluster). Note that Runhouse OSS is facilitating the setup (via SkyPilot) and rpc, but users don't don't need to create a Runhouse account or anything like that, this is strictly inside their own cloud accounts with their own credentials (or using their IP and ssh creds without a cloud account).

## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [x] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).

Comments:
_The documentation is not available anymore as the PR was closed or merged._

Definitely, that makes sense, and thanks for the fast review! Updated per your suggestions.

Thanks for iterating! There is just the issue of the tests that are not running now. It seems there is an issue with your CircleCI permissions. Could you try refreshing your permissions as shown [here](https://support.circleci.com/hc/en-us/articles/360048210711-How-to-Refresh-User-Permissions-)?

Oh sorry! I think I've granted it access. Do I need to trigger anything on my side?

Probably an empty commit to re-trigger the CI.

Ok, so it looks like we just need a quick `make style` to fix the formatting on the added examples and we should be good to go.

Oops, thought I already ran via `make fixup`. Pushed!

Ok, last failures are fixed on main so merging. Thanks!

Thank you, Sylvain!!
fix: Change is_last chunk calc and add conditional break in chunk_iter
Author: connor-henderson
Pull request: yes

# What does this PR do?
Fixes #21568 

With three functional changes
- Updates the is_last calc to include the right_stride
- Conditionally breaks at the end of the loop block if is_last is true
- Adds to a test

And two non-functional changes
- Renamed `i` to `chunk_start_idx` and put `chunk_end_idx` in a variable
- Removed a comment and added another


## Before submitting
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
[Issue](https://github.com/huggingface/transformers/issues/21568)
- [x] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [x] Did you write any new necessary tests?

## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

@Narsil


Comments:
_The documentation is not available anymore as the PR was closed or merged._

@ArthurZucker Could you please review this ?

I am ok with this PR, but since I made the proposed change, I'm too heavily biased to review properly.

I will take another look still now that I might be able to think straighter now.


Just thinking we should update the values of the whisper timestamps (so try running the slow tests for the ASR pipeline, only those related to whisper)

> Just thinking we should update the values of the whisper timestamps (so try running the slow tests for the ASR pipeline, only those related to whisper)

Updated the timestamps after running the below slow asr pipeline tests:
`test_return_timestamps_in_preprocess`
`test_torch_whisper`
`test_find_longest_common_subsequence`
`test_whisper_timestamp_prediction` (only update was here)
`test_simple_whisper_asr`
`test_simple_whisper_translation`

Thank you very much for this ! And detecting this bug !


Thanks again for your contribution!
Adding NLLB-200 - MoE - 54.5B for no language left behind
Author: PierreColombo
Pull request: no

### System Info

Hello  @LysandreJik,
Thanks a lot for your work on no language left behind.

Is there any plan to add the 54.4B Model?

Kindest regards


### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

Improvement

### Expected behavior

Improvement

Comments:
WDYT @ArthurZucker @younesbelkada given your work on MoEs?

Sure, we can add this to the to dos, @PierreColombo could you add the link to the open sourced checkpoints? 

Hi Thanks for your positive answer.

Code is here: https://github.com/facebookresearch/fairseq/tree/nllb

Checkpoints are here : https://tinyurl.com/nllb200moe54bmodel

Thanks !


Hi all,
This would be greatly appreciated!
Thanks

also cc @sheonhan re. NNLB

+1, would love to see it!

+1 here.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

unstale?

We went for the fairseq implementation :'(

Friendly ping @ArthurZucker 

Yes! @sheonhan mentioned wanting to take this, otherwise will gladly sprint !

Since I'm working on the Image Completion Transformer at the moment, I might be blocking the folks who want to use it asap, so you should go ahead! @ArthurZucker
[WIP]`NLLB-MoE` Adds the moe model
Author: ArthurZucker
Pull request: yes

# What does this PR do?
Fixes #21300
To-Dos:
- [x] Conversion script and original weights available [here](https://huggingface.co/ArthurZ/fairseq-nllb-moe)
- [x] Converted checkpoints and configuration file available:
        - [moe-128](https://huggingface.co/ArthurZ/nllb-moe-128) experts
- [x] Make the common tests go green
- [x] Implement top 2 gating mecanism
- [x] Add integration tests for:
    - [x] the routers
    - [x] the logits
    - [x] the generation using greedy search
  - [x] Cleanup the PR

Comments:
_The documentation is not available anymore as the PR was closed or merged._
TF: XLA beam search + most generation-compatible models are now also XLA-generate-compatible
Author: gante
Pull request: yes

# What does this PR do?

The much-awaited PR -- beam search is now XLA compatible. GPT2 is the only model with XLA beam search tests, more models will follow in subsequent PRs 🎊  Preliminary tests on my machine shows that XLA beam search on GPU is ~26x faster (greedy search and sample are ~30x faster).

Slow tests have been run for the usual generate models (gpt2, t5, rag, speech_to_text, encoder_decoder, vision_encoder_decoder, bart).

EDIT: I've also generalized a few functions, and now ALL models that are compatible with generate are also compatible with XLA generate (with a few exceptions, when the models have no past cache support)

__________________________________

A hard-earned lesson which is kinda obvious in hindsight: `if` branches can make the XLA compiler confused about variable shapes, tagging their shape as `<unknown>`, which in turn causes all sorts of exceptions. Out of curiosity, I tried replacing the `if` by `tf.cond`, but the `<unknown>` shape persisted (because the tensor could indeed have a different shape at tracing time, depending on the branch taken)





Comments:
_The documentation is not available anymore as the PR was closed or merged._

Very cool! Can we try it out for at least on Encoder-Decoder architecture as well (just to know that this code holds true here)?

@patrickvonplaten @Rocketknight1 now with encoder-decoder tests, and ready for review -- I was working on it on a separate branch, so I've merged it into this one. Now, this PR standardizes the XLA model kwargs preparation, and most models can use the XLA functionality. Some models were incompatible for different reasons, so there is a new flag to gate XLA generation (and the flag is set in the problematic architectures).

Finally, I'm also considering adding a general test like `test_xla_generate_fast`, but with `@slow`, beam search, and >100 tokens. It will probably break for a few models (like T5), but at least we would be able to automatically track which models are reliable with XLA beam search -- WDYT?  

Note: as per the comment above, if this PR gets merged as it is, I will open an issue to track issues regarding XLA generation (relevant models failing fast tests, as well as models failing the slow tests)
[Benchmarks] index
Author: stas00
Pull request: no
Labels: name: Benchmarks

name: WIP

This issue is to document the important `transformers` benchmarks in one place, so that they are easy to find.

To add a new benchmark entry post it in an Issue (separately or as a comment in an existing issue) and then link from here. If you have edit rights please add a link directly to this post, otherwise please add a note in the comments and I will update this post. 

Please do not post actual benchmarks in the comments of this Issue. This is only an index. 

Thank you!


## Fastest speed combinations
- [RTX-3090](https://github.com/huggingface/transformers/issues/14608#issuecomment-1005229426)
- [A100](https://github.com/huggingface/transformers/issues/15026#issuecomment-1005227577)


## Precision: fp16 vs bf16 vs tf32 vs fp32
- [RTX-3090](https://github.com/huggingface/transformers/issues/14608#issuecomment-1004390803)
- [A100](https://github.com/huggingface/transformers/issues/15026#issuecomment-1004543189)


## Batch size / gradient accumulation steps 
-  gradient accumulation steps: [RTX-3090](https://github.com/huggingface/transformers/issues/14608#issuecomment-1004392537), [A100](https://github.com/huggingface/transformers/issues/15026#issuecomment-1004592231)
- batch size [RTX-3090](https://github.com/huggingface/transformers/issues/14608#issuecomment-1004470417), [A100](https://github.com/huggingface/transformers/issues/15026#issuecomment-1005033957)
 


## Gradient checkpointing
- [RTX-3090](https://github.com/huggingface/transformers/issues/14608#issuecomment-1004422281)
- [A100](https://github.com/huggingface/transformers/issues/15026#issuecomment-1005034578)


## Optimizers:
- Adam torch vs. apex vs HF vs adafactor: [RTX-3090](https://github.com/huggingface/transformers/issues/14608#issuecomment-1005219385), [A100](https://github.com/huggingface/transformers/issues/15026#issuecomment-1005220263)
- re-run the above a year later with the same list of optimizers, plus BNB's 8bit optimizer and fused torch AdamW [PCIe 80GB A100](https://github.com/huggingface/transformers/issues/22101)

## Network / Interconnects:
- [DP/DDP/NVLink](https://github.com/huggingface/transformers/issues/9371#issuecomment-768656711)


Comments:
Possibly Incorrect Perplexity Calculation in Conceptual Guide
Author: fpgaminer
Pull request: no

In the docs, Conceptual Guides->Perplexity, the code underneath the section titled "Example: Calculating perplexity with GPT-2 in 🤗 Transformers" might be wrong.  This is based on my understanding.  The specific line of example code:

https://github.com/huggingface/transformers/blob/e8cc02555ee7dce7213e624ab088d8d4d1952064/docs/source/en/perplexity.mdx?plain=1#L122

I believe it should be:

`neg_log_likelihood = outputs.loss * (trg_len - 1)`

This is because `outputs = model(input_ids, labels=target_ids)` calculates `trg_len - 1` losses (and then averages them), not `trg_len`.

You can see why in the model code:

https://github.com/huggingface/transformers/blob/68287689f2f0d8b7063c400230b3766987abf18d/src/transformers/models/gpt2/modeling_gpt2.py#L1100C5-L1106

Basically, the HF API states "Note that the labels are shifted inside the model".  Because of this design choice, the model can only ever calculate `n - 1` losses, since it has to shift the labels itself.  `shift_logits = lm_logits[..., :-1, :].contiguous()` throws away the last logit, which it can't use to calculate a loss because it isn't given a label for the last position.  `shift_labels = labels[..., 1:].contiguous()` throws away the useless first label.

It's an odd decision in the API and should perhaps be a separate feature request to fix.  Regardless, this bug report is about the Perplexity calculation guide.  Since the model is only calculating `trg_len - 1` losses, it should only multiply the loss by `trg_len - 1`, not `trg_len`.

I believe the last line would also be wrong for similar reasons: `ppl = torch.exp(torch.stack(nlls).sum() / end_loc)`

An alternative is to change the guide to calculate the loss itself. This would allow the full `trg_len` labels to be used correctly.

Comments:
That's correct, would you like to open a PR with your fix?

I can, yes. Which would be best: a fix for those two lines, or re-writing to calculate loss outside the model?  The latter better matches how the guide explains things but is a more extensive change and requires updating verbiage in the guide.

Additionally, is it preferred to keep the existing style of multiplying the loss inside the loop and dividing outside?  Seems like a simple `mean` outside the loop is sufficient, shouldn't result in any reduction in numerical accuracy, and is more efficient.

I think the simple fix is enough. We can also average the losses outside of the loop indeed.
Fix bug in perplexity guide calculations and update perplexity numbers. Fixes #22348
Author: fpgaminer
Pull request: yes

# What does this PR do?

This pull request fixes the bug mentioned in issue #22348 relating to the Perplexity concept guide in the English documentation.

It removes the multiplication by `trg_len` (which was incorrect), and uses `mean` outside the loop instead of sum and division.  This both fixes the bug and simplifies the code.

I've updated the comments in the example code to reflect the changes, as well as leaving a note about the sub-optimal nature of the code.

N.B. This pull request does not touch the documentation in other languages besides English.

N.B. These changes are based on my understanding of perplexity and how the Transformers library works.  I'm not an expert.

Finally, the perplexity numbers were updated based on the new code.  I ran the example on my machine to get those new numbers.  My setup:

```
PyTorch Version: 2.0.0
GPU: NVIDIA GeForce RTX 3090
CUDA Version: 11.7
CUDA Device Capability: (8, 6)
CUDA Toolkit Version: 11
Transformers Version: 4.27.2
```


Fixes #22348


## Before submitting
- [X] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [X] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

@sgugger 


Comments:
_The documentation is not available anymore as the PR was closed or merged._