noamaanMulla-03/transformers-issues huggingface dataset:
[HELP] Secure Delivery of Trained LLM for Client Demo
Author: humza-sami
Pull request: no
Labels: name: Usage

I have recently trained a LLM based on llama-2 using a private dataset for a client. They require the model for a demo on their machines. Unfortunately, I don't have the option to host the model and provide them with an endpoint. The model size is  approximately 68GB, and it's stored in SafeTensors. Additionally, I have developed a binary for inference+RAG pipeline. I am using vLLM for inference. The model is located within a folder . The client needs to test the demo on their local machines.
I am seeking advice on the best possible secure method to deliver the LLM model to the client while ensuring that the model files are encrypted to prevent misuse. Given the sensitivity of the model and its potential misuse, encryption is crucial to maintain data security.

Comments:
Hey! Thanks for opening. I am sorry I don't think there is a specific answer I can give, would recommend to post this on the [forum](https://discuss.huggingface.co/)! instead! 🤗
[docs] Indent ordered list in add_new_model.md
Author: windsonsea
Pull request: yes

- Indent 3 spaces for ordered list items
- Fix some minor errors about backticks and grammar

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_29796). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Added HelpingAI model type in it
Author: OEvortex
Pull request: no
Labels: name: New model

### Model description


Introduction
HelpingAI-3B is a state-of-the-art AI model designed to assist with day-to-day tasks. It's trained on a diverse range of datasets, making it versatile and adaptable to various applications.

Model Overview
HelpingAI-3B is the latest model in the HelpingAI series. It's built on advanced machine learning algorithms and trained on a wide variety of data sources. This ensures that the model is capable of understanding and generating responses in a wide range of contexts.

Usage code
```python
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer, TextStreamer

# Let's bring in the big guns! Our super cool HelpingAI-3B model
model = AutoModelForCausalLM.from_pretrained("OEvortex/HelpingAI-3B", trust_remote_code=True, torch_dtype=torch.bfloat16).to("cuda")

# We also need the special HelpingAI translator to understand our chats
tokenizer = AutoTokenizer.from_pretrained("OEvortex/HelpingAI-3B", trust_remote_code=True, torch_dtype=torch.bfloat16)

# This TextStreamer thingy is our secret weapon for super smooth conversation flow
streamer = TextStreamer(tokenizer)

# Now, here comes the magic! ✨ This is the basic template for our chat
prompt = """

system: {system}



user: {insaan}



assistant:
"""

# Okay, enough chit-chat, let's get down to business!  Here's what our system will say to the user
system = "You are an adaptive and versatile AI assistant, ready to help with various topics and situations while maintaining a conversational, engaging, and friendly tone. You aim to provide accurate, comprehensive information and advice. Be open to feedback and adjust your responses based on user input. Always show empathy and understanding in your conversations."


# And the insaan is curious (like you!) insaan means user in hindi
insaan = "Hey HelpingAI, how's it going?"

# Now we combine system and user messages into the template, like adding sprinkles to our conversation cupcake
prompt = prompt.format(system=system, insaan=insaan)

# Time to chat! We'll use the tokenizer to translate our text into a language the model understands
inputs = tokenizer(prompt, return_tensors="pt", return_attention_mask=False).to("cuda")

# Here comes the fun part!  Let's unleash the power of HelpingAI-3B to generate some awesome text
generated_text = model.generate(**inputs, max_length=3084, top_p=0.95, do_sample=True, temperature=0.7, use_cache=True, streamer=streamer)
```

### Open source status

- [X] The model implementation is available
- [X] The model weights are available

### Provide useful links for the implementation

_No response_

Comments:
Hey! Could you link a potential paper / github repo we can refer too? 🤗
Disable AMD memory benchmarks
Author: IlyasMoutawwakil
Pull request: yes

# What does this PR do?

This PR disables the memory measurements using `pynvml`/`py3nvml` on AMD GPUs as `nvml` is not compatible with ROCm.

Benchmark tests passing: https://github.com/huggingface/transformers/actions/runs/8431639514/job/23095964749

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_29871). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Set custom_container in build docs workflows
Author: Wauplin
Pull request: yes

**TL;DR:** this PR doesn't change anything but prevents future break.

Related to https://github.com/huggingface/doc-builder/pull/487. There is now a `custom_container` option when building docs in CI. When set to `""` (instead of `"huggingface/transformers-doc-builder"` by default), we don't run the CI inside a container, therefore saving ~2min of download time. The plan is to test disabling the transformers container on a few "big" repo and if everything works correctly, we will stop making it the default container. More details on https://github.com/huggingface/doc-builder/pull/487. This PR explicitly set the custom_container to transformers's one to avoid any breaks.

cc @mishig25 

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_29855). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Thanks for reviewing :)
Fix 29807, sinusoidal positional encodings overwritten by post_init()
Author: hovnatan
Pull request: yes

# What does this PR do?

Fixes huggingface/transformers#29807, sinusoidal positional encodings overwritten by post_init(). First time contributing, please let me know of any issues, comments.

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
      https://github.com/huggingface/transformers/issues/29807
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [x] Did you write any new necessary tests?


## Who can review?
@ArthurZucker, @younesbelkada, @amyeroberts



Comments:
@ArthurZucker `_init_weights()` is applied to every module in the model. I moved sinusoidal positional encoding generation after `post_init()` in my latest commit. Is that ok?

It does but you can check if the name is `"position_embeddings"`! 🤗 

AFAIK `post_init()` uses pytorch `Module`'s `apply(init_func)` functionality which doesn't pass name of the module to the `init_func()`, see https://pytorch.org/docs/stable/generated/torch.nn.Module.html#torch.nn.Module.apply.

Anyways, please look at my latest changes. They solve the problem, but not sure if this is optimal
Reimplement "Automatic safetensors conversion when lacking these files"
Author: LysandreJik
Pull request: yes

PR https://github.com/huggingface/transformers/pull/29390 was reverted as it was failing in users' runtime.

Two past issues:
- https://github.com/huggingface/transformers/issues/29500
- https://github.com/huggingface/transformers/issues/29513

This PR enables it with a slight change: all exceptions can be caught and ignored so as to leave the user's runtime alone.

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_29846). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
backward compatibility of GPT2Model class
Author: JohnHerry
Pull request: no

### System Info

Platform: Linux CentOS7 (in docker)
Python: 3.9.7
transformers: 4.38.2


### Who can help?

@ArthurZucker and @younesbelkada

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

I am using the GPT2Model class inside the transformers.

1、Install transformers==4.24.0
2、train a GPT2Model object, eg.  abc.pt
3、In other environment, install the transformers==4.38.2,
`
state_dict=torch.load("abc.pt")
model = GPT2Model(....)
mymodel = model.load_state_dict(state_dict["model"])
` 
4、step3 will raise Exception in 4.38.2 transformers,  while no such problem in 4.24.0 transformers

RuntimeError: Error(s) in loading state_dict for GPT2Model:
       Unexpected key(s) in state_dict: "model.h.0.attn.bias", "lm.model.h.0.attn.masked_bias", "model.h.1.attn.bias", "model.h.1.attn.masked_bias", ..... "model.h.11.attn.bias", "model.h.11.attn.masked_bias"

also, when I train the model in transformers==4.38.2 and load in 4.24.0,  it will report lossing such keys.


### Expected behavior

GPT2Model should better to be version compatible with transformers version update.

Comments:
Hey! Why are you not using the `from_pretrained` API, which is the recommended way to load a model and it's weights? 

We are loading a checkpoint of certain steps. not the final result.
And in fact, The GPT2Model worked as a main module in our network.

I can help you if the way you are loading the checkpoint is conventional but would recommend you to simply load with from_pretrained, and save it again 

> I can help you if the way you are loading the checkpoint is conventional but would recommend you to simply load with from_pretrained, and save it again

Thank you for the help.  Can I make decision that the warning parameters  "model.h.0.attn.bias", "lm.model.h.0.attn.masked_bias", "model.h.1.attn.bias", "model.h.1.attn.masked_bias", ..... "model.h.11.attn.bias", "model.h.11.attn.masked_bias" are irrelevant and that they can be removed from the parameter dict when I load checkpoint with torch.load?

Yes! The `bias` are `register_buffer` in later transformers release.  They can be safely ignored

Thank you very much.
LLaMa-VID: An Image is Worth 2 Tokens in LLMs
Author: amyeroberts
Pull request: no
Labels: name: New model

### Model description

LLaMA-VID is an open-source chatbot trained by fine-tuning LLaMA/Vicuna on GPT-generated multimodal instruction-following data. LLaMA-VID empowers existing frameworks to support hour-long videos and pushes their upper limit with an extra context token. We build this repo based on LLaVA.

LLaMA-VID contains three parts: encoder and decoder are adopted to produce visual embedding and text-guided features, respectively; context token and content token are transformed with the tailored token generation strategy; instruction tuning is designed to unleash the potential of LLMs for image and video.

### Open source status

- [X] The model implementation is available
- [X] The model weights are available

### Provide useful links for the implementation

Page: https://llama-vid.github.io/
Weights already available on HF: https://huggingface.co/YanweiLi/llama-vid-7b-pretrain-224
Code: https://github.com/dvlab-research/LLaMA-VID

Comments:
Hi @amyeroberts , Can I take up the task of porting this model  ?


@venkateshtata Certainly! Feel free to open a PR and ping me when it's ready for review. Let us know if you have any questions about porting into transformers in the meantime :) 

@amyeroberts ,

Its been couple of months there is no activity.
Can I give a try ? 

Sure! We prioritise reviewing based on when PRs are open, rather than comments on issues, as we find this prevents them from becoming too stale. You're free to open a PR to address any issue with no linked PR, or with a PR which is old and hasn't been reviewed / isn't active

@amyeroberts  ,
I am facing some issues to re-implement the results.  Should the authors of the paper be responsible to assist ? 

@nileshkokane01 I wouldn't say responsible - after all they're not asking us to add the model! 

Who to ask depends on what the issue is. If you're unable to replicate numbers just following the instructions on the [model's README](https://github.com/dvlab-research/LLaMA-VID?tab=readme-ov-file#install), then I would reach out to them, probably by opening an issue on the repo first. 

If however it's an issues when porting the original model to transformers, it's best to ask us on a draft PR of the port so we can see the code. 

@amyeroberts ,
Thanks, that helps .
vocab_size for modeling_llama.py
Author: SeunghyunSEO
Pull request: no

### System Info

transformers==4.39.0

### Who can help?

-

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

-

### Expected behavior

[this line](https://github.com/huggingface/transformers/blob/main/src/transformers/models/llama/modeling_llama.py#L1218) should use `self.vocab_size` not `self.config.vocab_size`

```python
shift_logits = shift_logits.view(-1, self.vocab_size)
```

because one can resize the embedding matrix for faster training (padded_vocab_size).
i think you guys consider this (`self.vocab_size = getattr(config, "padded_vocab_size", config.vocab_size)`) but forgot to use.


Comments:
mistral also has same problem

The api to resize the model usually also update the `config.vocab_size` so not 100% sure this affects anyone appart people who manually resize. But no, `padded_vocab_size` is not part of llama config. 

Using slef.vocab_size is equivalent I feel, but maybe more consistent. You can open a PR if you want!

thank you for the kind reply @ArthurZucker , i think official llama or mistral ckpt does not have problem like you mentioned but in our project, we override and customize llama like model and got probem. im not sure it is because config should be updated but it didn't... however i think we all agree to change the line to use self.vocab_size for potential issue! i will PR for the mistral and llama soon :)

Thanks 🤗 

Sorry for the misunderstanding, Arthur.
After checking the code a bit more, it seems that this only exists in the arguments that are generated when converting the Megatron model. I looked up the history of hugginface transformers' and there is code that uses padded_vocab_size, but it seems to be a very rare case. Also, as I mentioned, it seems that users rarely pre-pad the embedding of mistral and llama models outside of the trainer, and in that case, it would be the user's fault for not modifying the config, so I think it does not need to change the code.
Sorry for wasting your time..!

No worries 🤗
How to finetune a language model after extent token embeddings?
Author: bluewanderer
Pull request: no

If I add some new tokens for a language model, I will get some random initialized weights in embeddings and lm_head. Is there any official way to train only these new weights? Or all I can do is adding hooks to the tensors to zero the gradient for weights I do not want to change?

Comments:
Hey 🤗  thanks for opening an issue! We try to keep the github issues for bugs/feature requests.
Next time try to ask this kind of general questions on the [forum](https://discuss.huggingface.co/) instead? I'm sure the community will be of help!

If you only want to train new weights you should freeze the reste, official way is [to use torch.](https://discuss.pytorch.org/t/how-do-i-freeze-the-specific-weights-in-a-layer/104722)

Initializing the weights should be done [via this tutorial to not be fully random. ](https://nlp.stanford.edu/~johnhew/vocab-expansion.html)

Thanks!
Gate jitter is missing from Sparse MoE routing implementation
Author: kalomaze
Pull request: no
Labels: name: Good Second Issue

name: Feature request

### System Info

N/A

### Who can help?

@ArthurZucker 

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

In the official Mixtral MegaBlocks fork, there is a mention of [gate jitter.](https://github.com/mistralai/megablocks-public/blob/15634e007ecfff64cf19b4eefccdb5de4c1fe7c9/megablocks/layers/router.py#L40)

If you look at [modeling_mixtral.py](https://github.com/huggingface/transformers/blob/39ef3fb248ba288897f35337f4086054c69332e5/src/transformers/models/mixtral/modeling_mixtral.py), you can note that there is no mention of gate jitter, which serves as a regularization technique for MoE routing (I believe it is also described as "[Noisy gating](https://paul-bruffett.medium.com/mixture-of-experts-419806770255)" or in the context of [the original Sparse MoE paper](https://arxiv.org/pdf/1701.06538.pdf), as "Noisy Top-K gating".)

The jitter is implemented elsewhere in the repository for [Switch Transformers](https://github.com/huggingface/transformers/blob/704b3f74f9685e1772acd9949f65b0c5bbd64539/src/transformers/models/switch_transformers/modeling_switch_transformers.py#L172), but it is completely absent in the sparse MoE implementation provided for Mixtral, despite it being implemented in the official MistralAI MegaBlocks fork.

### Expected behavior

I believe that `MixtralSparseMoeBlock` needs to be adjusted so that the forward method includes a call to a new jitter method (similar to the one in the LearnedRouter class of the Megablocks implementation) that applies gate jitter to the input data before passing it through the MoE layer, if it is turned on.
It is unclear whether or not the jitter was used in the first place to pretrain Mixtral, but this would bring it to feature parity.

Comments:
would you like to open a PR for this? 🤗
MixtralSparseMoeBlock: add gate jitter
Author: lorenzoverardo
Pull request: yes

# What does this PR do?

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

This PR adds gate jitter to MixtralSparseMoeBlock's input data before passing it through the MoE layer, if turned on.
Fixes https://github.com/huggingface/transformers/issues/29429.

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [X] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [X] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [X] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

- text models: @ArthurZucker and @younesbelkada



Comments:
Thanks for the contribution!

FYI @younesbelkada 

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_29865). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Add early stopping callback to pytorch trainer
Author: cbrochtrup
Pull request: yes

# Summary

Address PyTorch half of https://github.com/huggingface/transformers/issues/4894 by adding early stopping patience and a minimum threshold metrics must improve to prevent early stopping. I piggybacked heavily off of https://github.com/huggingface/transformers/pull/7431/ since the two functions are very similar.

Since https://github.com/huggingface/transformers/pull/4186 seems to be abandoned and behind master, I figured I'd take a crack at this.

## Who can review?

Anyone! But @julien-c and @sgugger seem the most appropriate.

Comments:
Hi there. Thanks your PR! When I was designing the callbacks, it was to be them small independent pieces of code. I would prefer if early stopping had its own callback that the user would then choose to add or not. Do you think you could amend your PR in that direction?

Hello, thank you for your feedback! I will amend the PR in that direction.

Could you clarify which pieces of early stopping should be in `TrainerState` and which should be in the callback? I'm grappling with the similarities between `best_model_checkpoint` and early stopping attributes.

```python
class EarlyStoppingCallback(TrainerCallback):
    best_metric: Optional[float] = None # maybe not this
    best_model_checkpoint: Optional[str] = None # maybe not this either
    early_stopping_patience: int = None
    early_stopping_patience_counter: int = None

    def on_evaluate(self, args, state, control, **kwargs):
        # Keep track of patience
        # End training via early stopping
        if (
            self.early_stopping_patience is not None
            and self.early_sotpping_patience_counter >= self.early_stopping_patience
        ):
            control.should_training_stop = True
```

Or do you mean I just move the if statement I added to its own callback and keep `TrainerState` as is?

The `TrainerState` shouldn't change, so the callback you are writing above sounds fine, without the arguments marked with `# maybe not this`, which should already be in the `TrainerState`, I think.
Does that sound right to you?

That makes sense. I think [this](https://github.com/huggingface/transformers/blob/e812753736f475b62849ef0e72149306408c1395/src/transformers/trainer.py#L910) block of code (to line 933) could be a callback because it's all about the best metric. Then users could customize the best model calculations. Is that desirable?

If you think that's out of scope I'll keep the early stopping callback simple and separate from the best metric calculation.

I had put it in `Trainer` because I thought multiple callbacks could need it and it's used by `load_best_model_at_end` which is kind of a core feature.

Sounds good, you know best! I keep `load_best_model_at_end` in the `Trainer` and push up an early stopping callback sometime this week.

Thanks for your thorough and affable review!