noamaanMulla-03/transformers-issues huggingface dataset:
Add a warning when the dataset has no matching columns
Author: Thytu
Pull request: no
Labels: name: Feature request

### Feature request

Currently, when a user provides a dataset to `Trainer` without any columns matching the parameters of the `model.forward` method, an `IndexError` is raised without any accompanying warning or logs to aid the user in understanding this behavior. This lack of feedback can lead to confusion and increased debugging time, as experienced and discussed in [this thread](https://discuss.huggingface.co/t/trainers-dataloader-influenced-by-target-model/75685).

I suggest adding a warning log when such a scenario occurs. Additionally, considering whether directly raising an error could be an option might be beneficial. However, it's important to consider cases where a user could intentionally provides a dataset with no matching columns.

### Motivation

As a user, I wasn't initially aware that columns not accepted by the `model.forward()` method are automatically removed during the creation of the dataloader. Consequently, I spent several hours debugging my dataloader, model, and dataset without understanding why I kept receiving an `IndexError`. Implementing a warning log could potentially save users from similar debugging efforts in the future.

### Your contribution

I'm ready to create a pull request! :D

Comments:
Hey! do you want this to live in the trainer? cc @muellerzr 

Hi! Yes exactly, I was thinking about something like that: [link to commit change](https://github.com/huggingface/transformers/compare/main...Thytu:transformers:add-warning-when-dataset-has-no-matching-columns)
Add support for models trained using MLX
Author: alexweberk
Pull request: no

### Feature request

It would be great if model weights trained and saved through MLX could be easily loaded using the Transformers library. This way, MLX users and Transformers users can more freely collaborate when making open source models.


### Motivation

Currently, Transformers only supports safetensors files with metadata for format in `["pt", "tf", "flax"]`. If Transformers can add `"mlx"` to the list, and if MLX can add `metadata={"format":"mlx"}` when saving safetensors, this can be achieved.

Here is a related issue I created on the `mlx_lm` side:
https://github.com/ml-explore/mlx/issues/743#issuecomment-1965427589

Here is a related PR on the `mlx_lm` repo: https://github.com/ml-explore/mlx-examples/pull/496

### Your contribution

I can send a PR to modify the exact code shortly after.

Comments:
As additional motivation: this would enable fine-tuning locally with MLX -> save in safetensors -> load model in transformers which would be 🔥 

fyi @pcuenca 

Thanks for the ping @ArthurZucker! I'm supportive. Just opened https://github.com/huggingface/transformers/pull/29511 to further examine all the implications. I forgot to add @alexweberk as a co-author as he already identified the path to follow, I'll do it in another commit (or feel free to take over @alexweberk and lead the work)!
test_generation_config_is_loaded_with_model  - fall back to pytorch model for now
Author: amyeroberts
Pull request: yes

# What does this PR do?

Tests started failing because the new safetensors weights added for a test model on https://hf.co/joaogante/tiny-random-gpt2-with-generation-config

Failure message:
```
FAILED tests/test_modeling_utils.py::ModelUtilsTest::test_generation_config_is_loaded_with_model - ValueError: weight is on the meta device, we need a value to put in on 0.
```

Example circleci run: 
https://app.circleci.com/pipelines/github/huggingface/transformers/86552


This PR forces the test to used to old, pytorch weights whilst a fix is found. 

cc @gante 

Comments:
@Rocketknight1 great minds ;) 

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_29521). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Set `inputs` as kwarg in `TextClassificationPipeline`
Author: alvarobartt
Pull request: yes
Labels: name: Feature request

# What does this PR do?

This PR replaces the current `*args` arg within `TextClassificationPipeline` with a kwarg named `inputs` to be able to use it as a keyword arg when e.g. trying to unpack a Python dict as `pipeline(**{"inputs": "text"})` which is not possible with the current approach since the `*args` were being used instead.

Additionally, the usage of a kwarg aligns the `TextClassificationPipeline` implementation with the rest of the `Pipeline` subclasses, and not being a breaking change, since before no keyword arg could be specified.

## Before submitting

- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [x] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [x] Did you write any new necessary tests? -> There was no need to write any test, since the unpacking of `*inputs` works the same way as `*args` did, only adding the possibility to use it as a kwarg too.

## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

Library:

- pipelines: @Rocketknight1 

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_29495). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

> This seems like a good PR to me (with one formatting nit), and I agree that using `*args` for the sole argument is limiting. We've seen other issues where users want to pass extra keyword args to `TextClassificationPipeline`, such as #29271 - do you think this PR could resolve that issue as well?

Hmm I think that issue is referring to the fact that the input args under `**kwargs` are being "sanitized" so that just the recognized ones are used, while the `forward` method may accept more than those for their specific use case, so not sure this will be covered anyway with this PR, but maybe the `_sanitize_parameters` function could be refactored to validate the "known" ones, while leaving the rest of those there up to the users awareness and propagate those to the `forward` pass of that model? Not sure though, probably could break some UX.

@alvarobartt makes sense! Either way, I'm happy to approve this. cc @ArthurZucker for core maintainer review!
Fix `VisionEncoderDecoder` Positional Arg
Author: nickthegroot
Pull request: yes

# What does this PR do?

When using certain models as the encoder for `VisionEncoderDecoder` (e.g. `LayoutLMv3`), `pixel_values` could incorrectly be the model's first positional argument (`input_ids` for `LayoutLMv3`).

This PR fixes the issue by specifying `pixel_values` as a keyword argument and adds the appropriate tests to validate going forward.


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [X] Did you write any new necessary tests?


## Who can review?

@amyeroberts

Comments:
@amyeroberts fixed in 69c2385 & ready to merge!

@nickthegroot Code quality checks are currently failing - can you run `make fixup` and push the changes?

> @nickthegroot Code quality checks are currently failing - can you run `make fixup` and push the changes?

@amyeroberts - whoops, fixed in 406469f!
Unexpected type casting from `torch.bfloat16` to `torch.float32`
Author: pietrolesci
Pull request: no

Hi there,

I am loading a `BertForSequenceClassification` model in (true) `bfloat16` precision. I noticed that when I manually implement the steps in the [forward pass](https://github.com/huggingface/transformers/blob/a0857740c0e6127485c11476650314df3accc2b6/src/transformers/models/bert/modeling_bert.py#L1543) (i.e., get the pooler output, apply dropout, and pass it through the classifier head) the logits type is, as expected, `bfloat16`. However, if I simply pass a batch in the forward method and then get the logits, their type is now `float32`. In code,

```python
inp_ids, att_mask = batch["input_ids"], batch["attention_mask"]

model(input_ids=inp_ids, attention_mask=att_mask).logits.dtype
#>>> torch.float32

pooled_out = model.bert(input_ids=inp_ids, attention_mask=att_mask).pooler_output
pooled_out = model.dropout(pooled_out)
model.classifier(pooled_out).dtype
#>>> torch.bfloat16
```

This seems unexpected. I think the issue is due to the implementation of the `SequenceClassifierOutput` https://github.com/huggingface/transformers/blob/49204c1d37b807def930fe45f5f84abc370a7200/src/transformers/modeling_outputs.py#L1010-L1013

Comments:
Hey! I doubt the type hint influence the casting of the inputs. Also many things can play into this, the output is better in float32 if you are doing softmax afterward, see Llama for example

Hi @ArthurZucker, you are right: I thought the dataclass was doing some automatic typecheck, but it's not

```python
import torch
from transformers.modeling_outputs import SequenceClassifierOutput

loss = torch.tensor([1.], dtype=torch.bfloat16)
SequenceClassifierOutput(loss=loss).loss.dtype == torch.bfloat16
#>>> True
```

I agree that float32 is more desirable as an output type. However, the inconsistency between the two implementations (i.e., default forward and "manual" forward) is a bit confusing. Happy for you to close this issue if you think this is not confusing for other users :)

Alright, closing then! It's the same for Llama models, unless you manually reproduce every single operation! 🤗
New transformers module `4.37.2` saving the fine-tuned model as model.safetensors
Author: pratikchhapolika
Pull request: no

### System Info

I recently encountered an unexpected behavior while using the latest `Transformers package version 4.37.2.` 
Tf2.9, Python3.9

Upon training a model and attempting to save it using the `Trainer module`, instead of the expected `pytorch_model.bin` file, the model is being saved as `model.safetensors`. 

This departure from the usual saving convention has raised questions regarding whether this is a **new feature** introduced in the Transformers package or a feature of the underlying PyTorch package.


Questions:

Is the saving behavior observed (model.safetensors instead of pytorch_model.bin) a newly introduced feature in the Transformers package or a characteristic of the PyTorch package?

How can I ensure that both `model.safetensors and pytorch_model.bin` are saved concurrently while using the Trainer module for model fine-tuning and saving?



### Who can help?

@ArthurZucker @sgugger 

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

training_args = TrainingArguments(
        output_dir="v4_1",  # output directory
        num_train_epochs=10,  # total number of training epochs
        per_device_train_batch_size=8,  # batch size per device during training
        per_device_eval_batch_size=8,  # batch size for evaluation
        warmup_steps=(len(X_train)*6)//80,  # number of warmup steps for learning rate scheduler
        weight_decay=0.1,  # strength of weight decay
        logging_steps=500,
        evaluation_strategy='epoch',
        save_strategy = "epoch",
        logging_dir="logs",
        do_eval=  True,
        fp16=True,
        load_best_model_at_end = True,
        dataloader_num_workers = 4,
        #metric_for_best_model = "eval_precision",
        greater_is_better = "False",
        learning_rate = 1e-5
    )

trainer = Trainer(
        model=model,  # the instantiated 🤗 Transformers model to be trained
        args=training_args,  # training arguments, defined above
        train_dataset=train_dataset,  # training dataset
        eval_dataset=val_dataset, # evaluation dataset
        compute_metrics=compute_metrics # function to compute metrics for model eval
    )

trainer.train()
trainer.save_model("model_new")

### Expected behavior

save pytorch_model.bin

Comments:
Hey! This is part of our big refactoring that makes sure we save our models as `safetensors` vs `.bin` files which can execute arbitrary code. 

Inviting you to read [the doc](https://huggingface.co/docs/safetensors/en/index) here and [the corresponding argument to `save_pretrained`](https://huggingface.co/docs/transformers/main_classes/model#transformers.PreTrainedModel.save_pretrained.safe_serialization).

TLDR; it's expected

> the corresponding argument to `save_pretrained`

Perfect!! Thank you for replying! Answers my Question.
Quantization Error: register_buffer is not in module_parameters.
Author: DonggeunYu
Pull request: no

### System Info

Not a system issue. Code issue.

### Who can help?

_No response_

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

Code:
~~~
import torch
from transformers.models.deformable_detr.modeling_deformable_detr import DeformableDetrModel
from transformers import BitsAndBytesConfig


bnb_config = BitsAndBytesConfig(
        load_in_4bit=True,
        llm_int8_skip_modules=[
            'deformable_detr_model',
        ],
        bnb_4bit_compute_dtype=torch.float32,
    )
model = DeformableDetrModel.from_pretrained("SenseTime/deformable-detr", device_map="cuda:0", torch_dtype=torch.half, quantization_config=bnb_config)
~~~


### Expected behavior

Log:
~~~
Traceback (most recent call last):
  File "/selfsup/a.py", line 13, in <module>
    model = DeformableDetrModel.from_pretrained("SenseTime/deformable-detr", device_map="cuda:0", torch_dtype=torch.half, quantization_config=bnb_config)
  File "/usr/local/lib/python3.10/dist-packages/transformers/modeling_utils.py", line 3502, in from_pretrained
    ) = cls._load_pretrained_model(
  File "/usr/local/lib/python3.10/dist-packages/transformers/modeling_utils.py", line 3926, in _load_pretrained_model
    new_error_msgs, offload_index, state_dict_index = _load_state_dict_into_meta_model(
  File "/usr/local/lib/python3.10/dist-packages/transformers/modeling_utils.py", line 802, in _load_state_dict_into_meta_model
    or (not hf_quantizer.check_quantized_param(model, param, param_name, state_dict))
  File "/usr/local/lib/python3.10/dist-packages/transformers/quantizers/quantizer_bnb_4bit.py", line 124, in check_quantized_param
    if isinstance(module._parameters[tensor_name], bnb.nn.Params4bit):
KeyError: 'bias'
~~~

Comments:
Sorry for the delay! 
cc @SunMarc and @younesbelkada 

This should be already solved in this [PR](https://github.com/huggingface/transformers/pull/29329). I think we can close the issue ! Thanks again @DonggeunYu for raising the issue and proposing a PR to fix the issue. 

> This should be already solved in this [PR](https://github.com/huggingface/transformers/pull/29329). I think we can close the issue ! Thanks again @DonggeunYu for raising the issue and proposing a PR to fix the issue.

Thank you~
added support for llama v2 and codellama in weight conversion for issue #28241
Author: christoukmaji
Pull request: yes

# What does this PR do?

This PR adds support for LLaMa V2 and CodeLLaMa while maintaining backwards compatibility for LLaMa V1 in the LLaMa-HuggingFace weight conversion script `src/transformers/models/llama/convert_llama_weights_to_hf.py`. This PR changes the max_position_embeddings for LLaMa V2 to 4096, and for CodeLLaMa to 16384, while maintaining a default max_position_embeddings of 2048 for LLaMa V1.



Fixes #28241 


## Who can review?

@ArthurZucker @amyeroberts 


Comments:
Hey @christoukmaji, thanks for opening the PR! Seems like  #28754 was opened a bit earlier so we'll try to get it merged! 🤗 

Hi @ArthurZucker, thanks for the response. I would like to avoid the duplication of work for future contributions.

What is the PR selection process for HuggingFace contributions; is it first comment or first PR? I thought it was the first comment as outlined in the [Contribution documentation](https://huggingface.co/docs/transformers/contributing) and how [other PR's have been handled](https://github.com/huggingface/transformers/issues/28265).

Hey, pretty sure that if you look at the PR, it's first PR first, then if there is no activity anyone can take it. 
I'll update the contribution guidelines as commenting is not really enough as we can't track the progress / if you are stuck of if you even started. Sorry for that! 🤗 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Generate: left-padding test, revisited
Author: gante
Pull request: yes

# What does this PR do?

Now that we understand why left-padding doesn't produce the exact same outputs (see [comment](https://github.com/huggingface/transformers/issues/25420#issuecomment-1775317535)), this PR revisits the left-padding test.

In a nutshell:
- no more retry loop, instead sets a proper error tolerance for the test
- adds an appropriate skip reason (when needed)
- removes many model-specific skips -- these models either pass the updated test or fall under a skip rule


Comments:
cc @zucchini-nlp -- this is a good example of us adding ugly flakiness-related loops because we didn't understand the cause for flakiness. Now that we understand that left-padding == not exactly the same results, we can write a clean test.

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_29515). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Typo in mlx tensor support
Author: pcuenca
Pull request: yes

# What does this PR do?

https://github.com/huggingface/transformers/pull/29406 recently added MLX tensor support to `BatchEncoding`. However, I think there's a typo in `_is_mlx` when importing the `mlx.core` module. 

## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

cc @Y4hL for verification


Comments:
Yeah, it's a typo

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_29509). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Merged `main` to pull 4ed9ae623 and fix the failing test.

Feel free to merge whenever (I can't do it) :)
Early stopping can not save best model at last
Author: ILG2021
Pull request: no

### System Info

- `transformers` version: 4.35.0
- Platform: Linux-5.15.0-88-generic-x86_64-with-glibc2.31
- Python version: 3.10.11
- Huggingface_hub version: 0.17.3
- Safetensors version: 0.4.0
- Accelerate version: 0.24.1
- Accelerate config: 	not found
- PyTorch version (GPU?): 2.0.1 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: <fill in>
- Using distributed or parallel set-up in script?: <fill in>

### Who can help?

@sanchit-gandhi

### Information

- [x] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

I finetune whisper with transformers and set a easy stopping in the trainer. The easy stop works but can not find best model:
the code is:
```
trainer = Seq2SeqTrainer(
    args=training_args,
    model=model,
    train_dataset=common_voice["train"],
    eval_dataset=common_voice["test"],
    data_collator=data_collator,
    compute_metrics=compute_metrics,
    tokenizer=processor.feature_extractor,
    callbacks=[EarlyStoppingCallback(early_stopping_patience=3)]
)
```

the error report is:

`Could not locate the best model at whisper-large-v2-1.1.8/checkpoint-800/pytorch_model.bin, if you are running a distributed training on multiple nodes, you should activate `--save_on_each_node`.`

I have check the checkpoint folders no pytorch_model.bin in there, but why the trainer try to find pytorch_model.bin?

![image](https://github.com/huggingface/transformers/assets/93691919/496abca7-9f4f-41de-9eac-6c3d12f60024)


### Expected behavior

the best model can been saved at last.

Comments:
cc @muellerzr 

Hi @ILG2021, what is in `trainer.args`?

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.