noamaanMulla-03/transformers-issues huggingface dataset:
Fix 'local_rank' AttiributeError in Trainer class
Author: mocobeta
Pull request: yes

# What does this PR do?

This PR fixes `AttributeError: 'Trainer' object has no attribute 'local_rank'`.
Please see the discussion at https://github.com/huggingface/transformers/pull/23681 for details.

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- text models: @ArthurZucker and @younesbelkada
- vision models: @amyeroberts
- speech models: @sanchit-gandhi
- graph models: @clefourrier

Library:

- flax: @sanchit-gandhi
- generate: @gante
- pipelines: @Narsil
- tensorflow: @gante and @Rocketknight1
- tokenizers: @ArthurZucker
- trainer: @sgugger

Integrations:

- deepspeed: HF Trainer/Accelerate: @pacman100
- ray/raytune: @richardliaw, @amogkam

Documentation: @sgugger, @stevhliu and @MKhalusova

HF projects:

- accelerate: [different repo](https://github.com/huggingface/accelerate)
- datasets: [different repo](https://github.com/huggingface/datasets)
- diffusers: [different repo](https://github.com/huggingface/diffusers)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Maintained examples (not research project or legacy):

- Flax: @sanchit-gandhi
- PyTorch: @sgugger
- TensorFlow: @Rocketknight1

 -->


Comments:
This works for me with python 3.10.12 (Google Colab runtime).

cc @muellerzr 

_The documentation is not available anymore as the PR was closed or merged._

Friendly ping @muellerzr
[WIP] Add LaVIN model
Author: rishabbala
Pull request: yes

# What does this PR do?

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

Fixes (issue)


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- text models: @ArthurZucker and @younesbelkada
- vision models: @amyeroberts
- speech models: @sanchit-gandhi
- graph models: @clefourrier

Library:

- flax: @sanchit-gandhi
- generate: @gante
- pipelines: @Narsil
- tensorflow: @gante and @Rocketknight1
- tokenizers: @ArthurZucker
- trainer: @sgugger

Integrations:

- deepspeed: HF Trainer/Accelerate: @pacman100
- ray/raytune: @richardliaw, @amogkam

Documentation: @sgugger, @stevhliu and @MKhalusova

HF projects:

- accelerate: [different repo](https://github.com/huggingface/accelerate)
- datasets: [different repo](https://github.com/huggingface/datasets)
- diffusers: [different repo](https://github.com/huggingface/diffusers)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Maintained examples (not research project or legacy):

- Flax: @sanchit-gandhi
- PyTorch: @sgugger
- TensorFlow: @Rocketknight1

 -->


Comments:
Getting Sentence level log probabilities using this model
Author: Shashi456
Pull request: no
Labels: name: Discussion

name: wontfix

So I was trying to implement the SLOR score, using the transformer-XL, mostly avoiding training. But given the XL model and the sentence, how could i go about getting the sentence level log probability?


Attached is the formula I'm trying to implement.
![image](https://user-images.githubusercontent.com/18056781/56021217-81d6f700-5d26-11e9-8c46-9b88f0277398.png)


Thanks a lot for your help.


Comments:
So p_M(S) is just the output of the model right?

For p_u(S), I think the easiest is probably to use the empirical probabilities.
`TransfoXLTokenizer` has a counter to store words frequencies [here](https://github.com/huggingface/pytorch-pretrained-BERT/blob/master/pytorch_pretrained_bert/tokenization_transfo_xl.py#L98) which should be populated in the "pretrained" tokenizer so I would use and normalize this to get unconditional probabilities for each word and then compute SLOR.

@thomwolf So the code would look something like : 
```sentence = "Where is the dog going"
s_ids = convert_sentence_to_ids(sentence)
config = TransfoXLConfig()
model = TransfoXLModel(config)
last_hidden_state, new_mems = model(input_ids)
for i in sentence : 
     uni_prob =  freq(token)/no_of_tokens
     sent_uni_prob * = uni_prob 
SLOR = (1/len(sentence))* last_hidden_state - sent_uni_prob 
```

or am i mistaken about the idea that the last_hidden_state is the sentence probability i'm looking for

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


In case someone found this, I found an implementation that seems legit here https://github.com/sgondala/GoogleConceptualCaptioning/commit/981e1b61ca5f84052f6237402319714d7fe70b80.
Decoding adds space between special tokens when skip_special_tokens = True
Author: Praful932
Pull request: no

### System Info

- `transformers` version: 4.30.2
- Platform: Linux-5.15.109+-x86_64-with-glibc2.31
- Python version: 3.10.10
- Huggingface_hub version: 0.15.1
- Safetensors version: 0.3.1
- PyTorch version (GPU?): 2.0.0 (True)
- Tensorflow version (GPU?): 2.11.0 (True)
- Flax version (CPU?/GPU?/TPU?): 0.6.9 (gpu)
- Jax version: 0.4.8
- JaxLib version: 0.4.7
- Using GPU in script?: no
- Using distributed or parallel set-up in script?: -

### Who can help?

 @ArthurZucker

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

```python
from transformers import AutoTokenizer, AutoModelForSeq2SeqLM

model_id = "lmsys/fastchat-t5-3b-v1.0"
tokenizer = AutoTokenizer.from_pretrained(model_id, use_fast = False)

inp = tokenizer.encode("Hello world")
print(inp)

out = tokenizer.decode(inp, skip_special_tokens = False,spaces_between_special_tokens=True)
print(out)
> 'Hello   world
'

out = tokenizer.decode(inp, skip_special_tokens = True,spaces_between_special_tokens=True)
print(out)
> 'Hello   world'

out = tokenizer.decode(inp, skip_special_tokens = True,spaces_between_special_tokens=False)
print(out)
> 'Hello world'


```

### Expected behavior

I expected the last two outputs to be same. In the 2nd last output, Since special tokens are skipped, no space should have been added even if `spaces_between_special_tokens=True`

Comments:
Hey! The behaviour is totally expected: you added `" "` as an added token. This means that it will not be split, and will not be encoded directly using the underlying model. 
Here is what happens: (check [this](https://github.com/ArthurZucker/transformers/blob/74cb2b1dd9755f6bd72fc9e65d1e374e287afd84/src/transformers/models/t5/tokenization_t5.py#L324-L327)

```python 
>>> tokenizer.encode("Hello world")
# 1. Split the input using a trie made of special tokens and added tokens: 
["Hello", " ", "world"]
# 2. Add a prefix space
[" Hello", " ", " world"]
# 3. Replace prefix space with meta-space
['▁Hello', ' ', '▁world']
# 4. Get the corresponding tokens
[8774, 32106, 296, 1]
>>> tokenizer.decode(inp, skip_special_tokens = False,spaces_between_special_tokens=True)
# When decoding, `convert_tokens_to_string` is called. A ` ` is added before every special token. But ` ` is a special token, so a space will be added to it
["▁Hello", "  ", "▁world"]
```
When you skip special tokens, `" "` is skiped, but you still have the space from the tokenizer that *joins*  the text on a space to properly reformat it.

You are not correctly using the API, I would recommend you to remove "  " from the special tokens.
Access to Transformers examples link broken . Impact on navigation as well
Author: SoyGema
Pull request: no

### System Info

### Context
Hello there! 👋
 I´m following [Translation](https://huggingface.co/docs/transformers/tasks/translation) Transformers tutorial . 
Thanks for making it possible! 
Currently running the script [run_translation.py](https://github.com/huggingface/transformers/blob/main/examples/tensorflow/translation/run_translation.py) and before changing transformers to version `4.31.0.dev` the following message appears          

https://github.com/huggingface/transformers/blob/5757923888246ea16b324f53c60ea444574005ed/src/transformers/utils/__init__.py#L218

When I follow the link , the following message appears. And when I click the _here_ link

<img width="1048" alt="Captura de pantalla 2023-06-26 a las 17 47 23" src="https://github.com/huggingface/transformers/assets/24204714/93afc7f8-ea28-4d19-b596-99120190fd21">

It redirects me to https://huggingface.co/docs/transformers/main/en/examples with an 404 error. 

### Potential Fix

Would love to give a helping hand here  🙏  like in #24336 and give back to the help I´ve gotten from #24254 but I am a little bit confused with respect to this. The last version in google-indexed examples that seem to work is [this](https://huggingface.co/docs/transformers/v4.15.0/examples) ,  related to `v4.15.0` and not `v4.30` nor `v4.29.2` .

Can you please confirm that you would validate this link
(https://huggingface.co/docs/transformers/v4.15.0/examples) for utils __init__.py script? If not, 
would you provide a useful link or point me in the right direction?

Please let me know if I'm also in the right place, as this could maybe impact website?
Thanks for the time dedicated to this.


### Who can help?

@sgugger @stevhliu

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

1. Go to link described in utils https://huggingface.co/docs/transformers/examples
2. Follow link provided [
](https://huggingface.co/docs/transformers/main/en/examples)  and find error 404

### Expected behavior

Last link to examples, with the stable version. Unclear to me at this point if it is `4.29` or `4.30` or where they are.

Comments:
The link is wrong, it should be https://huggingface.co/docs/transformers/run_scripts
Would you like to make a PR with the fix?

Sure, thanks!

BTW @sgugger @LysandreJik you can specify doc redirects using a `redirects.yml` like in datasets and other libraries: https://github.com/huggingface/datasets/blob/main/docs/source/_redirects.yml

Always good to avoid broken links when we can 

cc @mishig25 too
Update test versions on README.md
Author: sqali
Pull request: yes

# What does this PR do?

Hi @sgugger @amyeroberts , 

I have raised this PR is raised to improve the docs by updating the test versions mentioned in the README.md file. I have referred to the setup.py file to update it. It is related to issue #24263. I have followed the recommended documentation format as advised. Kindly check and advise. 

Fix #24263

Comments:
_The documentation is not available anymore as the PR was closed or merged._

Thanks, @sgugger. This is my first successful contribution to an open-source project.
Is time to update the transformers dependence in README.
Author: Lucas-Luoling
Pull request: no

In README docs, it say `This repository is tested on Python 3.6+, Flax 0.3.2+, PyTorch 1.3.1+ and TensorFlow 2.3+.`

Comments:
@luoling1993 Indeed! Would you like to open a PR to update this, so that you get the contribution? 

take

Hi @amyeroberts ,
As I can see this has not been fixed yet, I would love to raise a pull request to resolve this. May I know the latest versions it has been tested on? My apologies as I am new to this.

@sqali - great! Happy to hear you want to make this contribution. 

The supported versions can be found in [setup.py](https://github.com/huggingface/transformers/blob/main/setup.py).


Hi @amyeroberts,

I have made the following changes as per the setup.py file. However, I couldn't find PyTorch specifically mentioned, so I used Torch instead. Please review and kindly correct me if there are any mistakes. I have provided two formats below, and I would greatly appreciate your confirmation before I proceed with raising the pull request.

1.) "This repository has been tested with Python 3.7.0+, Flax >= 0.4.1 & <= 0.6.9, Torch >= 1.9 & != 1.12.0, and TensorFlow >= 2.4 & < 2.13."

2.) This repository is tested on the following versions:

- Python: 3.7.0+
- Flax: >= 0.4.1 & <= 0.6.9
- Torch: >= 1.9 & != 1.12.0
- TensorFlow: >= 2.4 & < 2.13

I kindly request your guidance and feedback regarding these changes. If you have any further suggestions or modifications, please let me know.

Thank you for your assistance.

@sqali Thanks for pulling this info. For the update, it's best to follow the current pattern in the docs: 

`This repository is tested on Python 3.7+, Flax 0.4.1+, PyTorch 1.9+ and TensorFlow 2.4+.`

Let's open a PR, and we can discuss any additional changes before merging there. 

Hi @amyeroberts ,
Thanks for the confirmation. Was a little skepitcal about the format. I will raise the PR now. 
Thanks for the assistance.

Hi @amyeroberts ,

I have raised the pull request and it has been approved by Sylvain Gugger. It has passed all the required checks and is ready to be merged. I would like to thank you for giving me the opportunity to raise the PR and make this contribution.

Thanks

Thanks for fixing and congrats on getting your first contribution merged in!
Generate: `group_beam_search` requires `diversity_penalty>0.0`
Author: gante
Pull request: yes

# What does this PR do?

While revisiting `group_beam_search` to review https://github.com/huggingface/transformers/pull/24407, I noticed that we do not require `diversity_penalty` to be `>0.0`. If it is not `>0.0`, then `group_beam_search` degenerates to `beam_search` with `num_beams=num_beams/num_beam_groups` -- users exploring the method risk not seeing its potential.

With this exception, we ensure the degeneration case does not happen (and possibly nudge the users towards the docs)

Comments:
_The documentation is not available anymore as the PR was closed or merged._
Generate: `min_tokens_to_keep` has to be `>= 1`
Author: gante
Pull request: yes

# What does this PR do?

As pointed out by @njhill in [this comment](https://github.com/huggingface/transformers/pull/24111#issuecomment-1601824441), `min_tokens_to_keep` has to be `>=1`. When it is not, the sampling step will lead to numerical exceptions, as all tokens have `-float("inf")` as logits.

This PR updates some of the checks, which were checking that it was `>=0`, and fixes the typical_p logits processor, which has the exact same issue as the one fixed in https://github.com/huggingface/transformers/pull/24111

Comments:
_The documentation is not available anymore as the PR was closed or merged._

Thanks @gante!
[Time-Series] time series patching, PatchTST
Author: elisim
Pull request: yes

This PR added a time series patching - PatchTST
Fixes https://github.com/huggingface/transformers/issues/22075

@kashif 

Kashif impl in gluonTS: https://github.com/awslabs/gluonts/pull/2748

Comments:
_The documentation is not available anymore as the PR was closed or merged._

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

comment

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

comment

PatchTST is in Gluon, thanks to @kashif . Closing here :) https://github.com/awslabs/gluonts/pull/2748
[Time-Series] Added link to the blog in Tips
Author: elisim
Pull request: yes

@kashif @NielsRogge 

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_24325). All of your documentation changes will be reflected on that endpoint.

file names changed, opened a new one here
https://github.com/huggingface/transformers/pull/24482
[i18n-<English>] Translating docs to <Chinese>
Author: liteli1987gmail
Pull request: no
Labels: name: WIP

<!--
Note: Please search to see if an issue already exists for the language you are trying to translate.
-->

Hi!

I translated all the English documents into Chinese.

Let's bring the documentation to all the <languageName>-speaking community 🌐 (currently 0 out of 267 complete)

Who would want to translate? Please follow the 🤗 [TRANSLATING guide](https://github.com/huggingface/transformers/blob/main/docs/TRANSLATING.md). Here is a list of the files ready for translation. Let us know in this issue if you'd like to translate any, and we'll add your name to the list.

Some notes:

* Please translate using an informal tone (imagine you are talking with a friend about transformers 🤗).
* Please translate in a gender-neutral way.
* Add your translations to the folder called `<languageCode>` inside the [source folder](https://github.com/huggingface/transformers/tree/main/docs/source).
* Register your translation in `<languageCode>/_toctree.yml`; please follow the order of the [English version](https://github.com/huggingface/transformers/blob/main/docs/source/en/_toctree.yml).
* Once you're finished, open a pull request and tag this issue by including #issue-number in the description, where issue-number is the number of this issue. Please ping @ArthurZucker, @sgugger for review.
* 🙋 If you'd like others to help you with the translation, you can also post in the 🤗 [forums](https://discuss.huggingface.co/).


<!--
Keep on adding more as you go 🔥
-->


Comments:
Please completely fill the template with your language and also search the existing GitHub issues to avoid duplicates.
Fix poor past ci
Author: ydshieh
Pull request: yes

# What does this PR do?

<img width="326" alt="Screenshot 2023-06-26 102340" src="https://github.com/huggingface/transformers/assets/2521628/461069ba-637a-4ebf-bb2e-103bba81bbcc">



:face_with_spiral_eyes: Let's be a bit nice to torch 1.11 and 1.10 🙏 .

(just a type issue: `(line 641)  RuntimeError: expected scalar type float but found double` introduced in #24334)

Comments:
_The documentation is not available anymore as the PR was closed or merged._
🌐 [i18n-KO] Translated `tflite.mdx` to Korean
Author: 0525hhgus
Pull request: yes

<!-- PR의 제목은 "🌐 [i18n-KO] Translated `<your_file>.mdx` to Korean" 으로 부탁드립니다 -->
# What does this PR do?

Translated the `tflite.mdx` file of the documentation to Korean 😄 
Thank you in advance for your review!

Part of https://github.com/huggingface/transformers/issues/20179
<!-- 메인 이슈에 기록이 남아요! 가짜연구소 리포를 사용해 연습하실때는 제거해주시면 감사하겠습니다! :smile: -->

## Before reviewing
- [x] Check for missing / redundant translations (번역 누락/중복 검사)
- [x] Grammar Check (맞춤법 검사)
- [x] Review or Add new terms to glossary (용어 확인 및 추가)
- [x] Check Inline TOC (e.g. `[[lowercased-header]]`)
- [x] Check live-preview for gotchas (live-preview로 정상작동 확인)

## Who can review? (Initial)

<!-- 1. 위 체크가 모두 완료된 뒤에만 가짜연구소 팀원들에게 리뷰 요청하는 아래 주석을 노출해주세요! -->
<!-- Team PseudoLab, may you please review this PR?  -->
@0525hhgus, @KIHOON71, @sim-so, @gabrielwithappy, @HanNayeoniee, @wonhyeongseo, @jungnerd

## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [x] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

## Who can review? (Final)

<!-- 2. 가짜연구소 팀원들과 리뷰가 끝난 후에만 허깅페이스 직원들에게 리뷰 요청하는 아래 주석을 노출해주세요! -->
May you please review this PR?
@sgugger, @ArthurZucker, @eunseojo

Comments:
_The documentation is not available anymore as the PR was closed or merged._

번역 문장 모두 좋습니다. 앞서 반영된 내용 외에 추가 의견 없습니다! 👍 

May you please review this PR? 😄 
@sgugger, @ArthurZucker, @eunseojo