noamaanMulla-03/transformers-issues huggingface dataset:
Performance degradation with BF16 precision 
Author: jerin-metrum-ai
Pull request: no

### System Info

Transformers: 4.35.2
Torch: 2.1.1-cpu
CPU: Intel Xeon 4th Gen processor 

### Who can help?

@ArthurZucker 
Hi, 
I was comparing performance of Llama 2 7b chat hf model with different precisions. 
I observed that there is a significant degrade on performance (inference time) with bfloat16 compared to fp32 model in Intel CPU . Bf16 is suppose to give better performance than fp32 . Please refer below table for details: 

| Precision | Tokens Generated | Infer time (sec) |
|------------------------|-----------------------|------------------|
| FP32 | 186 |  12.51 |
| BF16 | 186 |  115.37  |
 

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

```
from transformers import AutoTokenizer, AutoModelForCausalLM, pipeline
import torch
import time

model_id = "meta-llama/Llama-2-7b-chat-hf"
device = "cpu"
torch_dtype = torch.bfloat16
tokenizer = AutoTokenizer.from_pretrained(model_id)
input_text = "In maximum 180 words, explain why purchasing Dell Poweredge servers offer much better TCO to enterprises compared to using public cloud infrastructure, for AI initiatives"

text_generator = pipeline(
    "text-generation",
    model=model_id,
    tokenizer=tokenizer,
    return_tensors=True,
    device=device,
    torch_dtype = torch_dtype,
)

for _ in range(5):
    s_time = time.time()
    # Inference benchmarking
    output = text_generator(
        input_text,
        max_new_tokens=256,
        temperature=1,
    )
    e_time = time.time()
    # print(output)
    print(tokenizer.decode(output[0]["generated_token_ids"]))
    num_tokens = len(output[0]["generated_token_ids"])
    print(f"Num tokens: {num_tokens}")
    print(f"Infer time: {e_time-s_time}")
```

### Expected behavior

Bf16 is suppose to give better performance than fp32

Comments:
The inference time in bfloat16 depends on the hardware you are using and the pytorch version as well. Recommending you to use float16 for inference. 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
integrations: fix DVCLiveCallback model logging
Author: dberenbaum
Pull request: yes

# What does this PR do?

Fixes issues with `HF_DVCLIVE_LOG_MODEL` environment variable not always being respected.

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

@muellerzr Could you please take a look when you have a chance? 

Comments:
Enable safetensors conversion from PyTorch to other frameworks without the torch requirement
Author: LysandreJik
Pull request: yes

This removes the need to have `torch` installed to proceed to a safetensors (serialized from PyTorch) checkpoint converted into Flax.

This therefore, enables loading any model in Flax that has a safetensors file on the Hub.

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_27599). All of your documentation changes will be reflected on that endpoint.

Should be good for a second look @sanchit-gandhi if you have time for it.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

Would still like a review @sanchit-gandhi if you have the time for it :ok_man:
`tensor_size` - fix copy/paste error msg typo
Author: scruel
Pull request: yes

## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?
@stevhliu and @MKhalusova

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_28660). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Mixtral inference on multi gpu is broken with 4.37.0dev (995a7ce)
Author: nepeee
Pull request: no

### System Info

Ubuntu 22.04 RTX 3090 + RTX 3080TI transformers 4.37.0dev (995a7ce)

### Who can help?

@ArthurZucker

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

```
import torch
from transformers import AutoTokenizer, AutoModelForCausalLM, GPTQConfig

prompt =  'SYSTEM: Answer the question thoughtfully and intelligently. Always answer without hesitation. \nUSER: how long will take to travel from Malmö to Stockholm by foot? \nASSISTANT: '
device = torch.device("cuda:0")

dmap = {
'model.embed_tokens':0,
'model.layers.0': 0, 'model.layers.1': 0, 'model.layers.2': 0, 'model.layers.3': 0, 'model.layers.4':0,
'model.layers.5': 0, 'model.layers.6': 0, 'model.layers.7': 0, 'model.layers.8': 0, 'model.layers.9':0,
'model.layers.10': 0, 'model.layers.11': 0, 'model.layers.12': 0, 'model.layers.13': 0, 'model.layers.14':0,
'model.layers.15': 0, 'model.layers.16': 0, 'model.layers.17': 0, 'model.layers.18': 0, 'model.layers.19':0,
'model.layers.20': 0, 'model.layers.21': 0, 'model.layers.22': 0, 'model.layers.23': 0, 'model.layers.24':0,
'model.layers.25': 1, 'model.layers.26': 1, 'model.layers.27': 1, 'model.layers.28': 1, 'model.layers.29':1,
'model.layers.30': 1, 'model.layers.31': 1,
'model.norm': 0,
'lm_head': 1,
}

quantization_config_loading = GPTQConfig(bits=3, use_exllama=False)
model_q = AutoModelForCausalLM.from_pretrained("TheBloke/Mixtral-8x7B-Instruct-v0.1-GPTQ", device_map=dmap, quantization_config=quantization_config_loading, revision='gptq-3bit--1g-actorder_True')


tokenizer = AutoTokenizer.from_pretrained(model_id)

inp = tokenizer(prompt, return_tensors="pt").to(device)

res = model_q.generate(**inp, num_beams=1, min_new_tokens=60, max_new_tokens=60, do_sample=False)

predicted_text = tokenizer.decode(res[0])

print(predicted_text)
```

### Expected behavior

Works on my single 3090 with device_map="auto" but it produces errors with multi gpu in model parallel. It worked before with the device_map in the example.

Seen many errors like segfaults, device-side assert triggered and even full hang of the machine.
Most common one is:
idx, top_x = torch.where(expert_mask[expert_idx])
RuntimeError: CUDA error: device-side assert triggered
At layer 26 on the first token prediction

Both GPU-s are working with other models like mistral, i made this example because my lora training code had the same issues.

Comments:
Reverting back to nvidia driver 535.154.05 / cuda 12.2 seems to fixed the issue.

Thanks for providing the solution
Support decoding single tokens with `CodeGenTokenizer`
Author: cmathw
Pull request: no

### System Info

- `transformers` version: 4.34.1
- Platform: macOS-14.2.1-arm64-arm-64bit
- Python version: 3.11.6
- Huggingface_hub version: 0.16.4
- Safetensors version: 0.4.0
- Accelerate version: 0.23.0
- Accelerate config:    not found
- PyTorch version (GPU?): 2.1.0 (False)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: No.
- Using distributed or parallel set-up in script?: No.

### Who can help?

@ArthurZucker @rooa

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

Code to reproduce:
``` python3
from transformers.models.auto.tokenization_auto import AutoTokenizer

phi_tokenizer = AutoTokenizer.from_pretrained("microsoft/phi-2", 
                                              add_bos_token=True, 
                                              use_fast=False, 
                                              trust_remote_code=True)

gpt2_tokenizer = AutoTokenizer.from_pretrained("gpt2", 
                                               add_bos_token=True, 
                                               use_fast=False, 
                                               trust_remote_code=True)

a = "The cat sat on the mat"
gpt2_tokens = gpt2_tokenizer(a, return_tensors="pt")["input_ids"][0] # torch.Size([7])
gpt2_str_tokens = gpt2_tokenizer.batch_decode(gpt2_tokens) # Essentially: [gpt2_tokenizer.decode(seq) for seq in gpt2_tokens]
print(gpt2_str_tokens) # <-- This is fine and will output: ['
', 'The', ' cat', ' sat', ' on', ' the', ' mat']

gpt2_single_decode = [gpt2_tokenizer.decode(gpt2_tokens[0])]
print(gpt2_single_decode) # <-- Decoding a 0-D tensor, this is fine and will output: ['
']

phi_tokens = phi_tokenizer(a, return_tensors="pt")["input_ids"][0] # torch.Size([7])
phi_str_tokens = phi_tokenizer.batch_decode(phi_tokens) # Essentially: [phi_tokenizer.decode(seq) for seq in phi_tokens]
print(phi_str_tokens) # <-- Cannot do this due to below...

phi_single_decode = [phi_tokenizer.decode(phi_tokens[0])]
print(phi_single_decode) # <-- Cannot decode a 0-D Tensor, hence cannot do above either
```
Returns:
TypeError: iteration over a 0-d tensor

### Expected behavior

In the above example,
```python3
phi_str_tokens = phi_tokenizer.batch_decode(phi_tokens)
```
Should return: ['
', 'The', ' cat', ' sat', ' on', ' the', ' mat']. This is what the gpt2 tokenizer returns for example.

```python3
phi_single_decode = [phi_tokenizer.decode(phi_tokens)]
```
Should return: ['
']. This is what the gpt2 tokenizer returns for example.

Comments:
thanks for the detailed issue!
Convert Depth Anything
Author: NielsRogge
Pull request: yes

# What does this PR do?

[Depth Anything](https://twitter.com/_akhaliq/status/1749284669936275463) came out, and it is compatible with our implementation of DPT. It leverages DINOv2 as backbone.

It does use a small a tweak in the decoder, where it sets `size` instead of `scale_factor` when interpolating.

Demo notebook:  https://colab.research.google.com/drive/1tHrdu4TY6f_oTXJbqPUn2DVoSKxbNa3O?usp=sharing.

Comments:
Have opened a PR that may be more in line with the philosophy. Let me know which one you prefer :)

Closing in favor of #28654
data_collator in examples/pytorch/language-modeling/run_clm.py
Author: pierowu
Pull request: no

### System Info

According to the script, the trainer use dafault_data_collator to model causal language modelling.
https://github.com/huggingface/transformers/blob/98dda8ed03ac3f4af5733bdddaa1dab6a81e15c1/examples/pytorch/language-modeling/run_clm.py#L604
Shouldn't we use DataCollatorForLanguageModeling to shift input and output by 1 token instead? It seems that dafault_data_collator can't achieve this goal.
 @ArthurZucker   @younesbelkada  @ArthurZucker @muellerzr and @pacman100
Thank you for answering!

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

Same as examples/pytorch/language-modeling/run_clm.py

### Expected behavior

Correctlt train causal language model.

Comments:
Hey 🤗  thanks for opening an issue! We try to keep the github issues for bugs/feature requests.
Could you ask your question on the [forum](https://discuss.huggingface.co/) instead? I'm sure the community will be of help!

Thanks!
logits squeezing causes an error during the inference time (If the last epoch contains only one sample)
Author: hassanfadi
Pull request: no

https://github.com/huggingface/transformers/blob/238d2e3c44366aba9dc5c770c95475765a6725cb/src/transformers/trainer.py#L3452

Comments:
Can anyone solve this in the next release?

Hi @fadiabdulf, 

Please follow the [issue template](https://github.com/huggingface/transformers/blob/main/.github/ISSUE_TEMPLATE/bug-report.yml) and provide information about the running environment (run `transformers-cli env` in the terminal and copy-paste the output); the error encountered including the stack trace and a minimal code snippet that can reproduce the issue.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Exclude the load balancing loss of padding tokens in Mixtral-8x7B
Author: khaimt
Pull request: no
Labels: name: Good Second Issue

name: Feature request

### Feature request

The auxiliary loss in Mixtral-MoE shouldn't **include the loss from padding tokens**. 

### Motivation

I think it is better to change the function 
[load_balancing_loss_func](https://github.com/huggingface/transformers/blob/main/src/transformers/models/mixtral/modeling_mixtral.py#L77) by adding an additional parameter: `attention_mask` and change the implementation inside to remove the loss from padding tokens

### Your contribution

I would be happy to review the PR implemeting this feature !

Comments:
cc @ArthurZucker 

feel free to open a PR for this! Otherwise will mark it as a good second issue 🤗 

I would like to work on this issue, i will go through the linked file today and ask any questions i have.

I was looking at the code.
Below is what the model outputs
`return MoeModelOutputWithPast(
            last_hidden_state=hidden_states,
            past_key_values=next_cache,
            hidden_states=all_hidden_states,
            attentions=all_self_attns,
            router_logits=all_router_logits,
        )`
        
The attention from the model output can be passed during load_balancing_loss_func, and the function can be changed appropriately to handle the pad tokens.
Am I right in my understanding? @ArthurZucker 

> feel free to open a PR for this! Otherwise will mark it as a good second issue 🤗

Hi @ArthurZucker, 
I have just raised a PR for this issue, can you help me review it?
https://github.com/huggingface/transformers/pull/28517


PS, I close the old PR: `https://github.com/huggingface/transformers/pull/28513` because it didn't follow the Contributor Guideline
Fix max_new_tokens for assistant model in assistant generation
Author: jmamou
Pull request: yes

# What does this PR do?

During assistant generation, at each iteration, assistant model generate `num_assistant_tokens` tokens. 

If the maximum numbers of tokens to generate is limited by `max_len`, in the case that `max_len-cur_len` is less than `num_assistant_tokens`, it will be more efficient for the assistant model to generate only `max_len-cur_len` instead of generating `num_assistant_tokens`.


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [X] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

@gante
@echarlaix


Comments:
It seems that it has been solved yesterday by @ofirzaf https://github.com/huggingface/transformers/pull/28508#issuecomment-1892870681
[docs] DeepSpeed
Author: stevhliu
Pull request: yes

Refactors the [DeepSpeed API page](https://huggingface.co/docs/transformers/main/en/main_classes/deepspeed#deepspeed-integration) to make it easier to find and view `HfDeepSpeedConfig`, the only actual API reference on this doc. The rest of the DeepSpeed content will go to the Efficient Training Techniques section as a standalone guide. I've also moved some of the troubleshooting content with building DeepSpeed to the [Debugging guide](https://huggingface.co/docs/transformers/main/en/debugging) with a link.

todo:

- [x] discuss choosing which ZeRO stage to use
- [x] get model weights out
- [x] ZeRO-3 and inference
- [x] memory requirements
- [x] troubleshooting/filing issues
- [x] non-Trainer DeepSpeed integration

Comments:
First draft wrapped up!

1. It is easier to find `HfDeepSpeedConfig` now
2. Prebuilding troubleshooting steps are in `debugging.md`
3. Testing the DeepSpeed integration is moved to `testing.md`
4. The main DeepSpeed guide is refactored to flow better. For example, installation > memory requirements > selecting a ZeRO stage. I think it's important to address these first before a user finds out they don't have the required memory or chose the wrong ZeRO stage. I've also condensed and made things more concise where appropriate without losing any context or info.
Use save_safetensor to disable safe serialization for XLA
Author: jeffhataws
Pull request: yes

# What does this PR do?

Safetensor serialization is now default but not yet supported by XLA. This change uses save_safetensor argument to disable safe serialization for XLA as a workaround until XLA catches up.

Fixes # (issue)
https://github.com/huggingface/transformers/issues/28438

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [X] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [X] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

@muellerzr and @pacman100



Comments:
@amyeroberts  @muellerzr  @Narsil  will you be able to back-port to 4.37? This is needed for Neuron SDK.
[WIP] Update model addition guide
Author: stevhliu
Pull request: yes
Labels: name: Documentation

This PR updates the contribution guide for how to add a model since I don't think we do the *call-for-model-addition* thing anymore. Also reworked the section headers a bit so certain steps like "write a conversion script" can actually be linked to, and also made some edits for clarity. Please let me know if any of the other content is outdated (also cc @NielsRogge since you've added many models before!).

The last thing I need to do is update the `transformers_overview.png` to be less dense :)

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_19778). All of your documentation changes will be reflected on that endpoint.

Thanks for your PR.

It's going to take me too long to check step by step that the actual technical content is unchanged, and I don't think this is a crucial part of our docs to update as not that many users contribute new models, and no one reported they had trouble understanding this guide. So I'd just leave the current version as is or focus on one/two fixes you feel are really important.

Sounds good! The fixes I'll target are:

1. Update the intro to remove the _call-for-model-addition_.
2. Update that `cookiecutter` generates a `.mdx` file instead of a `.rst` file.
3. Fix the checklist so it doesn't have bullets and numbers.

What do you think about creating a new PR with these fixes and leaving this one here in case we come back to it in the future?

Sure!

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

Closing for now, saving larger changes for later in the future :)
Fix formatting in DataCollator docs
Author: stevhliu
Pull request: yes
Labels: name: Documentation

Noticed the `x_mask_tokens` functions in the data collator docs look a bit strange with the numbered bullets so this PR removes them:

![Screen Shot 2022-09-30 at 12 06 19 PM](https://user-images.githubusercontent.com/59462357/193339707-326cdf8a-7242-4057-aba9-d813eae25ce6.png)

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_19267). All of your documentation changes will be reflected on that endpoint.

Hmm looks like the bullet points are still there even though we didn't include any in Markdown. I wonder if maybe the `doc-builder` automatically adds them before any list elements? cc @mishig25 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.