noamaanMulla-03/transformers-issues huggingface dataset:
[BUG] `from_pretrained` does not properly initialize missing parameters under DeepSpeed ZeRO-3
Author: XuehaiPan
Pull request: no

### System Info

- `transformers` version: 4.36.2
- Platform: Linux-6.2.0-39-generic-x86_64-with-glibc2.35
- Python version: 3.11.5
- Huggingface_hub version: 0.20.1
- Safetensors version: 0.4.0
- Accelerate version: 0.25.0
- Accelerate config:    not found
- PyTorch version (GPU?): 2.1.2 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: True
- Using distributed or parallel set-up in script?: True

### Who can help?

@pacman100

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

Reproduce script:

```python
import deepspeed
import torch.distributed as dist
from transformers import AutoModelForSequenceClassification
from transformers.integrations.deepspeed import HfDeepSpeedConfig, is_deepspeed_zero3_enabled


def main() -> None:
    deepspeed.init_distributed()

    _hfdsc = HfDeepSpeedConfig(
        {
            'zero_optimization': {'stage': 3},
            'train_batch_size': 128,
            'train_micro_batch_size_per_gpu': 16,
            'gradient_accumulation_steps': None,
        },
    )
    assert is_deepspeed_zero3_enabled()

    model = AutoModelForSequenceClassification.from_pretrained('gpt2')

    with deepspeed.zero.GatheredParameters(params=[model.score.weight]):
        if dist.get_rank() == 0:
            print('weight', model.score.weight)


if __name__ == '__main__':
    main()
```

Commad line:

```console
$ torchrun --nnode 1 --nproc-per-node 8 test.py
...
[2023-12-25 23:49:31,983] [INFO] [partition_parameters.py:348:__exit__] finished initializing model - num_params = 149, num_elems = 0.12B
weight Parameter containing:
tensor([[0., 0., 0.,  ..., 0., 0., 0.],
        [0., 0., 0.,  ..., 0., 0., 0.]], device='cuda:0', requires_grad=True)
```

### Expected behavior

The parameters that are missing in the checkpoint should be randomly initialized.

These parameters are not initialized under the `_fast_init` setting:

https://github.com/huggingface/transformers/blob/fa21ead73db473d88f8eca1ec244aba776fd9047/src/transformers/modeling_utils.py#L3477-L3485

After the model successfully being partitioned under ZeRO-3, the parameter size is `torch.Size([])`. It will have no effect on the statement `model.apply(model._initialize_weights)`:

https://github.com/huggingface/transformers/blob/fa21ead73db473d88f8eca1ec244aba776fd9047/src/transformers/modeling_utils.py#L3995-L4005

Comments:
Using add_generation_prompt with tokenizer.apply_chat_template does not add the required assistant start token
Author: srikant86panda
Pull request: no

### System Info

Version:
transformers: 4.36.1 and transformers @ git+https://github.com/huggingface/transformers.git@5d36025ca13d05151b7a0c761e90d429c4644a30

Tokenizer: tokenizers==0.15.0

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

Running with 
```python
from transformers import AutoModelForCausalLM, AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("mistralai/Mistral-7B-Instruct-v0.2")

messages = [
    {"role": "user", "content": "What is your favourite condiment?"},
    {"role": "assistant", "content": "Well, I'm quite partial to a good squeeze of fresh lemon juice. It adds just the right amount of zesty flavour to whatever I'm cooking up in the kitchen!"},
    {"role": "user", "content": "Do you have mayonnaise recipes?"},

]
encodeds = tokenizer.apply_chat_template(messages, add_generation_prompt=False, tokenize=False)
print(f'without add_generation_prompt: {encodeds}')
encodeds = tokenizer.apply_chat_template(messages, add_generation_prompt=True, tokenize=False)
print(f'with    add_generation_prompt: {encodeds}')
```

### Expected behavior

With add_generation_prompt with tokenizer.apply_chat_template does should add the required assistant start token as discussed here: https://github.com/huggingface/transformers/issues/26539

Attached is a screenshot for reference
![Screenshot 2024-01-05 at 2 51 47 PM](https://github.com/huggingface/transformers/assets/18262494/87542f95-c259-47ae-abf0-61eefcaf1253)

Comments:
cc @Rocketknight1 

Hi @srikant86panda, Mistral and Mixtral both use the LLaMA prompt format, which does not include a special token at the start of assistant responses. Instead, they enforce that all conversations must alternate user/assistant/user/assistant, and the assistant response is always written immediately after the user message is finished with the `
` token.

As a result, `add_generation_prompt` does not have any effect for those models, and this is the correct and intended behaviour. I'm going to close this issue for now, but please let me know if you have any further problems or questions!

@Rocketknight1 thanks for clarifying it. My concern is good to be closed.
TGI Support for Mixtral AWQ
Author: RonanKMcGovern
Pull request: no

### Feature request

Currently, TGI seems to be able to load Mixtral AWQ models. However, the responses returned are blank.

### Motivation

It's possible to inference a Mixtral model from 16 bit weights (incl. with eetq if desired) but the downloading of weights is buggy and slow see [here](https://github.com/huggingface/text-generation-inference/issues/1413). Also, it would be nice to be able to just download 4-bit weights.

### Your contribution

The AWQ weights are good because they work in vLLM. So, there seems to be a bug on the TGI implementation (although it's unclear whether Mixtral is loading as a fluke because it doesn't seem to be explicitly supported for AWQ).

Comments:
Sorry, should have been posted in text-generation inference.

Also, TGI does work with AWQ. It's just that it doesn't work - at the time of writing with TheBloke's quant. See details [here](https://huggingface.co/TheBloke/Mixtral-8x7B-Instruct-v0.1-AWQ/discussions/1).
Fix test_modeling_mpt typo in model id
Author: JuanFKurucz
Pull request: yes

# What does this PR do?

Currently, the method `get_large_model_config` tries to download `mosaicml/mpt-7` model config, which is not found, rest of the file uses `mosaicml/mpt-7b`, which works. This seems like a typo.
## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

@ArthurZucker 


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_25606). All of your documentation changes will be reflected on that endpoint.
Enhancing Code Readability and Maintainability with Simplified Activation Function Selection.
Author: hisushanta
Pull request: yes

This code optimization enhances code readability and maintainability by utilizing aliases, simplified activation function selection, and consistent function definitions.

Before submitting
## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [x] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

## Who can Review?
@ArthurZucker 

Comments:
Fix bug of _prepare_4d_attention_mask
Author: jiqing-feng
Pull request: yes

Hi @ArthurZucker @younesbelkada . Since `_prepare_4d_attention_mask` is no longer a member function of `AttentionMaskConverter`, I directly import `_prepare_4d_attention_mask` from `modeling_attn_mask_utils`. Would you please help to review it? Thx!

BTW, the failed CIs are not related to my changes


Comments:
Hi @ArthurZucker @younesbelkada . All CIs are green, would you please help me review and merge it? Thx!
How to save the generated output of BarkModel to an npz file?
Author: chet-chen
Pull request: no

Hello there!

I'm using the BarkModel from Hugging Face Transformers and I'm wondering how to save the generated results to an npz file. I'd like to use these saved results as history prompts for the next generation.

In the [suno-ai/bark](https://github.com/suno-ai/bark) , when using the [`semantic_to_waveform`](https://github.com/suno-ai/bark/blob/main/bark/api.py#L35) method, I can pass `output_full = True`. This allows me to save the output to an npz file using `numpy.savez`.

However, as I transition to using the BarkModel within the transformers framework, I am uncertain about the equivalent process. Could you kindly provide guidance on how to save the generated results of the BarkModel to an npz file in the Transformers library?

Any assistance or code examples you could offer would be greatly appreciated.

Thank you for your time and support.

Comments:
cc @ylacombe 

Hey @chet-chen, let me take a look!
From the back of my mind, I think it requires to add something similar to `output_full = True` to BarkModel. It could be great that you do this if you add time, WDYT ?

@ylacombe I'm relatively new to Python, and I'm facing a challenge in converting the results generated by the BarkSemanticModel, BarkCoarseModel, and BarkFineModel models into a numpy.ndarray with the right shape. These models output torch.LongTensor, and I need help transforming them.

https://github.com/huggingface/transformers/blob/80377eb018c077dba434bc8e7912bcaed3a64d09/src/transformers/models/bark/modeling_bark.py#L1820-L1854

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
feat: add trainer label to wandb run upon initialization
Author: parambharat
Pull request: yes

# What does this PR do?
This PR adds a telemetry label to the wandb run making it possible to identify W&B usage from the trainer class.
 
ref: https://github.com/huggingface/transformers/pull/25590

## Before submitting
- [X] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?

## Who can review?
@LysandreJik 




Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_26466). All of your documentation changes will be reflected on that endpoint.

Thanks for your PR @parambharat!

Hi @LysandreJik , just checking in here. Can this be merged now ?

Hey @parambharat, I'd like either @muellerzr or @pacman100 to give it a look :)
fix(integrations): store serialized `TrainingArgs` to `wandb.config` without sanitization.
Author: parambharat
Pull request: yes

Allows resuming training runs when  reusing the wandb config.

# What does this PR do?

Currently the `WandbLogger` uses the `to_sanitized_dict()` method in `TrainingArguments` to serialize the training hyperparameters. This converts nested objects and `NoneType` objects to `str` for safe serialization.  However, using the stored config when resuming a training run leads to issues while initializing the `TrainingArguments` from the `wandb.run.config`. This PR fixes this by instead using the `to_dict` method to serialize the `TrainingArguments`. The resulting dictionary can be stored in the `wandb.config` and reused to resume training runs. 


## Before submitting
- [X] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests), Pull Request section?

## Who can review?


- trainer: @sgugger , @pacman100 


Comments:
_The documentation is not available anymore as the PR was closed or merged._
feature: update wandb callback to upload checkpoints
Author: parambharat
Pull request: yes

# What does this PR do?

The PR updates the `WandbCallback` with the following changes:
- Adds `on_save` method to upload model checkpoints as artifacts.
- Changes the default value of environment variable `WANDB_WATCH` from `gradients` to `false`. This enables quicker training when defaults are used. The user can easily change this behavior by setting the env variable.
- Changes the `WANDB_LOG_MODEL` variable from `bool` to `str` allowing for different settings to upload artifacts.
- Modifies the class dostring to reflect the above changes.
- Fixes broken link to wandb documentation
- Changes the wandb `run_name` from `output_dir` to wandb auto generated name. this avoids duplication of run names in wandb workspace

## Before submitting

- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [x] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).


## Who can review?
- trainer: @sgugger
- documentation: @stevhliu


## Examples
- Example [colab](https://colab.research.google.com/drive/17imujjBEL2cQL3odAJEbVvjR6zVsDRuH?usp=sharing) reflecting all the changes to the WandbCallback
- Example Weights & Biases [workspace](https://wandb.ai/parambharat/hf_transformers?workspace=user-parambharat) with runs that show different settings.
- Example Weights & Biases [Artifact](https://wandb.ai/parambharat/hf_transformers/artifacts/checkpoint/checkpoint-ajdcez6k/v4) created for checkpoints

Comments:
_The documentation is not available anymore as the PR was closed or merged._

Hey, @sgugger : Thanks for the quick review and suggestions. I've resolved all the issues. :hugs:

@parambharat There is a weird issue with the tests. Could you try refreshing your permissions as shown [here](https://support.circleci.com/hc/en-us/articles/360048210711-How-to-Refresh-User-Permissions-) and then pushing an empty commit?

Hey, @sgugger. I ran the fix that you suggested and all checks pass now.

Thank you @stevhliu. I've committed your recommendations.

Thanks again for your contribution!
Skip now failing test in the Trainer tests
Author: muellerzr
Pull request: yes

# What does this PR do?

https://github.com/huggingface/accelerate/pull/2319 introduced a revert on the DataLoader sampling logic to *not* use a SeedableRandomSampler by default as users were taken aback by the performance differences, so we've set it to `False` by default. This test is now back to its old way, where it was failing for ages. 

As noted in the `skip`, one of my next items to hit is a configuration for Accelerator that can be passed to the `TrainingArguments` that can customize this, but for now it's not the case. 

Fixes # (issue)


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

@ArthurZucker 


Comments:
[generation] Exact Search Decoding
Author: Saibo-creator
Pull request: no
Labels: name: Feature request

### Feature request

Hello Hugging Face Transformers Team,

I am writing to suggest a feature of an "exact search" decoding method, suggested in https://aclanthology.org/D19-1331/ 

Greedy search and beam search are both "greedy" in the sense that they are not guaranteed to find the global most likely generation. 

An exact search is a DFS-based search method with branch pruning that can guarantee to return the global optimal. 

The original implementation is located here: [DFS.py in SGNMT](https://github.com/ucam-smt/sgnmt/blob/master/cam/sgnmt/decoding/dfs.py).
Just need to adapt it to transformers generation module.



### Motivation

It has a strong research value because it returns the global optimal.
However, it may not be very practical for general users because it may be very slow.


### Your contribution

I could take the job to submit a PR if this is interesting for you. 
Otherwise, I can work on it as a fork under my account.

Comments:
FYI @gante 

Hi @Saibo-creator 👋 

I'll do my usual bargain! I haven't seen demand for this feature, so my current decision is negative. However, if this comment reaches 10 reactions within a 3-month time spawn, I'll revisit this decision -- it means the community is interested in the feature 🙌 

(whoever does the 10th reaction, please ping me)
Use mmap option to load_state_dict
Author: weimingzha0
Pull request: yes

# Use torch.load(mmap=True) to accelerate checkpoint loading

https://github.com/huggingface/transformers/issues/28332

cc @SunMarc @sgugger 








Comments:
> torch bin files are now deprecated in favor of safetensors but no harm in improving this! Coul you add a test as well in test_modelling_common? 🤗

Sure. Please let me know if my test case is correct.

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_28331). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

> Thanks! My last query would be to make sure this works for the deepspeed / sdpa case (device_map = "meta")!

Do you mean add a test case for device_map == "meta' ?

That would be a good way of making sure this will be fine with DeepSpeedZero (the if branch) 🤗 

> That would be a good way of making sure this will be fine with DeepSpeedZero (the if branch) 🤗

Currently none of existing tests covers the branch: 
I even added an assert in the "if branch" and all tests still passed except for one irrelevant error (see https://github.com/huggingface/transformers/pull/28401)

Anyway, I added a guard for "meta" device.
[Whisper] Fix slow test
Author: patrickvonplaten
Pull request: yes

# What does this PR do?

This PR fixes a slow test for Whisper that doesn't work anymore because it's using a deprecated dataset. For this slow test to pass, we will need the CI to have access to a security token (cc @ydshieh) 

Comments:
ah, you are too fast. See this comment 

https://github.com/huggingface/transformers/pull/28400#discussion_r1445856908

in short: `mozilla-foundation/common_voice_11_0` doesn't require token if this works for you

> mozilla-foundation/common_voice_11_0

The problem with CV 11.0 is that the expected results will change. 6_1 is exactly the dataset that was there before as well - is it that difficult. Don't we already have access to secret tokens? E.g. couldn't we just use `HUGGINGFACE_PUSH`? 

I personally don't know if `HUGGINGFACE_PUSH` is the right token (is it a HF token or github token).

We have

```
      token: ${{ secrets.HUGGINGFACE_PUSH }}
      hf_token: ${{ secrets.HF_DOC_BUILD_PUSH }}
```
Since we have such tokens, I think the infra team is fine with such tokens. We just need to talk to the infra team to make sure we are using the right one. And IMO, for read access, we should not use a push token (and better to have a read only token).

I will talk to the infra.

There is a new secret (token) added. I will update this PR accordingly and push

well, I have to add it to all workflow files 😅 

Thanks a lot @ydshieh! 

This PR should be good to merge!
[modelling] Missing causal mask in Llama model 
Author: KexinFeng
Pull request: no

### System Info

transformers-4.35.2


### Who can help?

@ArthurZucker @younesbelkd

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

Line https://github.com/huggingface/transformers/blob/2c658b5a4282f2e824b4e23dc3bcda7ef27d5827/src/transformers/models/llama/modeling_llama.py#L413C9-L413C9f

Here the causal mask is missing. See the gpt2 example in the expected behaviour below. Consequently, when using llama model to do inference, at the prefill step, the casuality is broken, i.e. the early tokens attends to the later ones. 

### Expected behavior

https://github.com/huggingface/transformers/blob/2c658b5a4282f2e824b4e23dc3bcda7ef27d5827/src/transformers/models/gpt2/modeling_gpt2.py#L194-L202

Comments:
Attention masks were recently refactored, see the https://github.com/huggingface/transformers/blob/2c658b5a4282f2e824b4e23dc3bcda7ef27d5827/src/transformers/models/llama/modeling_llama.py#L889 method.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.