noamaanMulla-03/transformers-issues huggingface dataset:
fix Starcoder FA2 implementation
Author: pacman100
Pull request: yes

# What does this PR do?
Fixes https://github.com/huggingface/transformers/issues/28889

With this PR, SDPA and FA2 gives almost similar outputs:
1. Code from the above issue:
```
import os

os.environ["CUDA_VISIBLE_DEVICES"] = "1"
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer


model_path = "bigcode/starcoderbase"

tokenizer = AutoTokenizer.from_pretrained(model_path)

dtype = torch.bfloat16  # also works for torch.float16
device_map = 'auto'  # also works for single GPU
model_flash = AutoModelForCausalLM.from_pretrained(model_path, device_map='cuda:0', attn_implementation="flash_attention_2", torch_dtype=dtype)
model_sdpa = AutoModelForCausalLM.from_pretrained(model_path, device_map=device_map, attn_implementation="sdpa", torch_dtype=dtype)
# sdpa and eadge give the same results, skip here
# model_eager = AutoModelForCausalLM.from_pretrained(model_path, device_map='cuda:0', attn_implementation="eager", torch_dtype=dtype)

def gen(prompt, model):
    model_inputs = tokenizer([prompt], return_tensors="pt").to(model.device)
    generated_ids = model.generate(**model_inputs, max_new_tokens=30, do_sample=False)
    text = tokenizer.batch_decode(generated_ids)[0]
    return text


print('=' * 50)
prompt = '<filename>spec.yaml'
print('[flash attention 2]')
print(gen(prompt, model_flash))
print('-' * 50)
print('[sdpa]')
print(gen(prompt, model_sdpa))

print('=' * 50)
prompt = "<fim_prefix>def fib(n):<fim_suffix>    else:\n        return fib(n - 2) + fib(n - 1)<fim_middle>"
print('[flash attention 2]')
print(gen(prompt, model_flash))
print('-' * 50)
print('[sdpa]')
print(gen(prompt, model_sdpa))
```

2. Output:
```
Loading checkpoint shards: 100%
7/7 [00:08<00:00, 1.01s/it]
Loading checkpoint shards: 100%
7/7 [00:08<00:00, 1.07s/it]
Setting `pad_token_id` to `eos_token_id`:0 for open-end generation.
==================================================
[flash attention 2]
Setting `pad_token_id` to `eos_token_id`:0 for open-end generation.
<filename>spec.yaml
name: "k8s-cluster"
version: "0.1.0"
summary: "Kubernetes cluster"
description:
--------------------------------------------------
[sdpa]
Setting `pad_token_id` to `eos_token_id`:0 for open-end generation.
<filename>spec.yaml
name: "k8s-cluster"
version: "0.1.0"
summary: "Kubernetes cluster"
description:
==================================================
[flash attention 2]
Setting `pad_token_id` to `eos_token_id`:0 for open-end generation.
<fim_prefix>def fib(n):<fim_suffix>    else:
        return fib(n - 2) + fib(n - 1)<fim_middle>
    if n < 2:
        return 1


--------------------------------------------------
[sdpa]
<fim_prefix>def fib(n):<fim_suffix>    else:
        return fib(n - 2) + fib(n - 1)<fim_middle>
    if n < 2:
        return n


```

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_28891). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Fix Keras scheduler import so it works for older versions of Keras
Author: Rocketknight1
Pull request: yes

See bug report [here](https://github.com/huggingface/transformers/pull/28588#issuecomment-1924842630)

This one caught me out - we used to import from `tf.keras.optimizers.schedules`, which worked on older versions of Keras. However, as part of the Keras 3 compatibility patch, we moved all `tf.keras` imports to `keras` imports, which normally should make no difference. However, in older versions of Keras the schedule classes actually live in `keras.optimizers.schedules.learning_rate_schedule` (even though they're in the right place in the equivalent version of `tf.keras`). I had no idea about this, and didn't catch it in my testing because I was using newer versions of TF!

Thanks to @echan5 for the warning!

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_28895). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Add custom loss to Informer
Author: ntakouris
Pull request: yes

# What does this PR do?

This PR adds a custom loss capability to the informer configuration and the informer model.
Currently, there is also the ability to specify an `nll` loss, but any loss of type `Callable[[torch.distributions.Distribution, torch.Tensor, float], torch.Tensor]` can be used.

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ x ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

@sgugger 
@elisim 
@kashif 



Comments:
Hey! Thanks for opening the PR, we try to leave things such as custom loss computation to the user (not providing the labels should do the trick) 🤗
Update the cache number
Author: ydshieh
Pull request: yes

# What does this PR do?

To git rid of the issues comes from torch stuff (torch vs torchaudio versions). 

See #28899

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_28905). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Yes, I thought changing cache number will fix all, but unfortunately no. There are some inter-dependencies prevent to get the desired torch/torchvision/torchaudio versions.

I don't have the time to look at this in depth - it took me quite time in the last two days. Let's unblock other PRs in a quick way and I will check later 🙏
[Docs] Fix placement of tilde character
Author: khipp
Pull request: yes

# What does this PR do?

This PR fixes the placement of the tilde character in examples for the internal link syntax.

## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).

## Who can review?

Documentation: @stevhliu

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_28913). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Force pushed since tests were not running.
Fix utf-8 yaml load for marian conversion to pytorch in Windows
Author: SystemPanic
Pull request: yes

# What does this PR do?

Fix yaml load for yaml files with UTF-8 encoding in convert_marian_to_pytorch.py

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed.

@ArthurZucker

Comments:
@ArthurZucker 

Yes, you can reproduce it with following Marian yaml file (Github doesn't allow .yml upload, so **change the file from .txt to .yml after download**):

```
with open('opusTCv20210807.spm32k-spm32k.vocab.yml') as f:
    print(yaml.load(f, Loader=yaml.BaseLoader))
```

and with the simple pull applied:

```
with open('opusTCv20210807.spm32k-spm32k.vocab.yml', encoding='utf-8') as f:
    print(yaml.load(f, Loader=yaml.BaseLoader))
```

[opusTCv20210807.spm32k-spm32k.vocab.txt](https://github.com/huggingface/transformers/files/14026656/opusTCv20210807.spm32k-spm32k.vocab.txt)


But I am not sure I follow why you are changing the file type? 

@ArthurZucker 

I already said it, Github has not included the .yml extension in their supported file types for the "Attach files" button.

You can check if you want, try to attach a .yml file in a Github comment, you can't.

Just renaming to .txt it's the simpler way to attach it. I don't get the point of confusion with this...

Sorry I got confused! 😓 alright thanks for providing a file, I can't reproduce your issue as it works out of the box for me

@ArthurZucker 

That's not possible. I forgot to mention that i'm using Windows 10.

transformers-cli env:

```
- `transformers` version: 4.37.1
- Platform: Windows-10-10.0.19045-SP0
- Python version: 3.10.11
- Huggingface_hub version: 0.19.4
- Safetensors version: 0.4.2
- Accelerate version: 0.22.0
- Accelerate config:    not found
- PyTorch version (GPU?): 2.1.0+cu118 (True)
- Tensorflow version (GPU?): 2.10.0 (True)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: No
```

OS default encoding is: Windows-1252 (CP-1252)

Thanks you!
[PatchTST] Some improvements
Author: NielsRogge
Pull request: yes

# What does this PR do?

When playing around with PatchTST, I've made some slight improvements:

- rename `target_values` to `labels` for the classification model
- rename bs->batch_size everywhere
- remove `num_targets` from the config and use `num_labels` instead

Comments:
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
torch.compile() silently fails when used on HuggingFace pipeline inference code
Author: rosario-purple
Pull request: no
Labels: name: WIP

### System Info

- `transformers` version: 4.35.2
- Platform: Linux-5.15.0-91-generic-x86_64-with-glibc2.35
- Python version: 3.10.13
- Huggingface_hub version: 0.19.4
- Safetensors version: 0.4.0
- Accelerate version: 0.25.0
- Accelerate config:    not found
- PyTorch version (GPU?): 2.1.1+cu121 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): 0.7.5 (cpu)
- Jax version: 0.4.21
- JaxLib version: 0.4.21
- Using GPU in script?: A100

### Who can help?

@Narsil @gante @ArthurZucker 

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

Run the following Python code:

```
    model = AutoModelForCausalLM.from_pretrained(
        MODEL_ID,
        torch_dtype=torch.bfloat16,
        device_map=device,
        use_flash_attention_2=True,
    )

    model.eval()

    tokenizer = AutoTokenizer.from_pretrained(MODEL_ID)
    tokenizer.pad_token_id = tokenizer.eos_token_id

    model = torch.compile(model)

    generation_pipeline = pipeline(
        "text-generation",
        model=model,
        tokenizer=tokenizer,
        batch_size=10,
    )

    batch_results = generation_pipeline(
        ["foo", "bar", "bin", "baz"],
        max_new_tokens=200,
        temperature=0.6,
        do_sample=True,
        repetition_penalty=1.05,
        num_return_sequences=20,
    )
```

(in my case, MODEL_ID is set to `"Open-Orca/Mistral-7B-OpenOrca"`, which is a fine-tune of Mistral-7B, but any LLM should work)

### Expected behavior

torch.compile() should compile the model, print some compilation messages, and then cause inference/text generation to be run faster. Instead, torch.compile() appears to not run at all, no messages are printed, and it has no effect on inference/generation speed. There is no error message, it just silently doesn't compile, effectively acting as if the line `model = torch.compile(model)` doesn't exist.

Comments:
No this can't work yet at we do not have a static KV cache. #27931 will fix this

While working on benchmarking and improving our model's performance I experienced the same issue with `torch.compile`. It does not bring any speed up at all and can confirm the issue. 

@ArthurZucker Looking forward to having `torch.compile` supported in transformers!

This is still broken

Indeed 🤗 marking as WIP until the PR is merged
Remove dead TF loading code
Author: Rocketknight1
Pull request: yes

This PR removes the dead `load_repo_checkpoint` method which as far as I know is unsupported and unused anywhere. This was discussed on Slack and because it's a potential security vulnerability and we don't even know what it could be used for anymore, we decided no deprecation cycle was needed

Comments:
I don't think those are major concerns - they're not in the public API, so the only way to exploit that code would be to get users to run those scripts manually. In general, I think `pickle.load` is fine in:

- The test suite
- Conversion scripts like these
- Scripts and examples

In all of those cases, attackers can't just modify a repo to insert malicious code that will get silently executed.

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_28926). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
fix: torch.int32 instead of torch.torch.int32
Author: vodkaslime
Pull request: yes

# What does this PR do?
Hi transformers newbie here. When reading the model codes, I found there might be a minor place where we could improve:

```
cu_seqlens = F.pad(torch.cumsum(seqlens_in_batch, dim=0, dtype=torch.torch.int32), (1, 0))
```
where the `dtype` is `torch.torch.int32`. Dunno why but wondering maybe it means `torch.int32`? So posting this PR trying to understand and to probably get it fixed if needed.

Thanks folks!

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

Comments:
@ArthurZucker  it's gonna be great if you could take a look if you have time.

 torch.torch.int32 is a redundant type hint indicating that both the first torch refers to the PyTorch library and the second torch refers to the int32 data type within that library. The actual conversion to int32 happens within torch.cumsum(), so the second torch is unnecessary.
cu_seqlens = F.pad(torch.cumsum(seqlens_in_batch, dim=0, dtype=torch.int32), (1, 0))


Failing is unrelated, merging ! Thanks @vodkaslime
-inf scores when generating with do_sample
Author: fColangelo
Pull request: no

### System Info

- `transformers` version: 4.37.1
- Platform: Linux-5.4.0-169-generic-x86_64-with-glibc2.29
- Python version: 3.8.10
- Huggingface_hub version: 0.20.3
- Safetensors version: 0.4.0
- Accelerate version: 0.24.1
- Accelerate config:    not found
- PyTorch version (GPU?): 1.14.0a0+44dac51 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: yes
- Using distributed or parallel set-up in script?: no

### Who can help?

@gante

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

To reproduce run the following script:
```
from transformers import AutoModelForCausalLM, AutoTokenizer
"""qconf = BitsAndBytesConfig {
  "bnb_4bit_compute_dtype": "bfloat16",
  "bnb_4bit_quant_type": "nf4",
  "bnb_4bit_use_double_quant": false,
  "llm_int8_enable_fp32_cpu_offload": false,
  "llm_int8_has_fp16_weight": false,
  "llm_int8_skip_modules": null,
  "llm_int8_threshold": 6.0,
  "load_in_4bit": true,
  "load_in_8bit": false,
  "quant_method": "bitsandbytes"
}"""

model = AutoModelForCausalLM.from_pretrained("mistralai/Mixtral-8x7B-Instruct-v0.1", quantization_config=qconf)
tokenizer = AutoTokenizer.from_pretrained("mistralai/Mixtral-8x7B-Instruct-v0.1")

input_ids = tokenizer.encode("
 
 Howdy! 
", return_tensors="pt")
out = model.generate(tokenizer.encode("
 
 Howdy! 
", return_tensors="pt"), 
                     max_length=4000, 
                     pad_token_id = tokenizer.eos_token_id,
                     return_dict_in_generate=True,
                     output_scores =True,
                     temperature=0.8,
                     top_k= 50,
                     do_sample= True)
print(out.scores)
```
out.scores will have almost only -inf values. The problem disappears when do_sample is removed.


### Expected behavior

logits are returned instead of -inf (happens when setting do_sample=False)

Comments:
Hi @fColangelo 👋 

That happens because `top_k` sets the scores of all but the `top_k` most likely tokens to `-inf` (50 in your case). If you want the model logits, and not the post-processed scores before sampling, have a look at this feature that is about to get merged: https://github.com/huggingface/transformers/pull/28667

Hi and thanks for the answer! I noticed this since I was trying to calculate a perplexity for an answer. I guess my code was incorrect, since the probability values for the selected tokens should not be -inf. I will check, confirm and close, ty!

It was in fact a bug in my indexing. Thanks for the info!
[Docs] Fix broken links and syntax issues
Author: khipp
Pull request: yes

# What does this PR do?

This PR fixes various external links that were broken due to syntax issues and updates several section links that specified incorrect anchor names.

It also corrects some comment tags and misspelled headings.

## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).

## Who can review?

Documentation: @stevhliu


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_28918). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
[WIP] Translate 'Contribute' Section into Chinese
Author: ghost
Pull request: no

Hello, I'm new here and this is my first time contributing. If there are any issues with my work, please let me know. I'll do my best to make any necessary changes.

This is a part of #26803 .
My plan is to translate the 6 sections of the 'Contribute' chapter into Chinese. The files that need translation include:

- [x] contributing.md #28243 
- [ ] add_new_model.md
- [ ] add_tensorflow_model.md
- [ ] add_new_pipeline.md
- [ ] testing.md
- [ ] pr_checks.md

There are interlinkages among these sections. For instance, in `contributing.md`, there are links referring to `testing.md`. Before translating `testing.md`, I'll temporarily link to the original language file. After translating `testing.md`, I'll update the links accordingly. Similar adjustments will be made for other interconnected files.I'll address these link adjustments at the end：

- [ ] Link adjustments for translated files

Looking forward to any feedback and suggestions!

Comments:
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Fix a wrong link to CONTRIBUTING.md section in PR template
Author: B-Step62
Pull request: yes

# What does this PR do?
The link https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests in the PR template is pointing to the deleted section. This PR replaces it to the correct one: https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request.

Fixes # (issue)


## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?
@stevhliu, @MKhalusova

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_28941). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.