noamaanMulla-03/transformers-issues huggingface dataset:
graphormer.collating_graphormer.preprocess_item TypeError: Cannot cast array data from dtype('int64') to dtype('int32') according to the rule 'safe'
Author: YueZhengMeng
Pull request: no

### System Info

- `transformers` version: 4.28.1                            
- Platform: Windows-10-10.0.19045-SP0                       
- Python version: 3.9.15                                    
- Huggingface_hub version: 0.10.1                           
- Safetensors version: not installed                        
- PyTorch version (GPU?): 1.13.1 (True)                     
- Tensorflow version (GPU?): not installed (NA)             
- Flax version (CPU?/GPU?/TPU?): not installed (NA)         
- Jax version: not installed                                
- JaxLib version: not installed                             
- Using GPU in script?: <fill in>                           
- Using distributed or parallel set-up in script?: <fill in>

### Who can help?

_No response_

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

The transformers.models.graphormer.collating_graphormer.preprocess_item cause TypeError: Cannot cast array data from dtype('int64') to dtype('int32') according to the rule 'safe'
When data_processed = dataset.map(preprocess_item, batched=False), the dataset is ogb-molhiv
Because the data type in this function is np.int64, but in transformers\models\graphormer\algos_graphormer.pyx is np.int32 or np.long
I fixed it by replace "long" in algos_graphormer.pyx line 88 and 89 by np.int64 ,and replace all 32 in algos_graphormer.pyx by 64
But I don't think this is a appropritate aproach


### Expected behavior

how to fix this issue without modifying the library code

Comments:
cc @clefourrier 

Hi @YueZhengMeng, thank you for reporting this!
I'll check it this week. Btw, using int64s everywhere will make the memory explode considerably faster for large graphs, if you need to change it manually in the mean time, it's likely you should use int32s instead.

I have not been able to reproduce your bug, but I'm not on a Windows machine.
Could you provide me with a snippet of code and the complete trace of your error?

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

Same issue. Also on windows. The autors solution worked for me. Change everything to 32 instead didn't work.
Support arbitrary processor
Author: ArthurZucker
Pull request: yes

# What does this PR do?
Saves the tokenizer class, making sure that the defaults that are stored in transformers are not blocking people from re-using pre-processors. Superseed #30030

For previous models:
```python3
import requests
from PIL import Image
from transformers import LlavaProcessor, LlavaForConditionalGeneration

checkpoint = "Intel/llava-gemma-2b"
processor = LlavaProcessor.from_pretrained(checkpoint, tokenizer_class="GemmaTokenizerFast")
```
But once saved, the tokenizer_class will be part of the `processor_config.json`, no need to specify it further.

Before merging: 
- [ ] Add tests
- [ ] Add doc


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_30875). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Thank you for adding this and I am so sorry for not following up!
Adding Special Tokens to GPT2 doesn't have any effect
Author: souravrai98
Pull request: no

### System Info


- `transformers` version: 4.40.2
- Platform: Linux-5.4.0-167-generic-x86_64-with-glibc2.31
- Python version: 3.12.3
- Huggingface_hub version: 0.23.0
- Safetensors version: 0.4.3
- Accelerate version: 0.30.1
- Accelerate config:    not found
- PyTorch version (GPU?): 2.3.0 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: No


### Who can help?

@Arthur @ArthurZucker 

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)


Comments:
Hello, thanks for opening an issue! We try to keep the github issues for bugs/feature requests.
Could you ask your question on the [forum](https://discuss.huggingface.co) instead?

Thanks!
Trainer using only one GPU instead of two
Author: oussaidene
Pull request: no

### System Info

transformers version 4.26.0
python version 3.8.8
pytorch version 1.9.0+cu102

### Who can help?

trainer: @sgugger, @muellerzr and @pacman100

### Reproduction

I am trying to train a T5 model using two gpus but for some reason the trainer only uses one?

in my bash file i specified the number of GPUs i wanna use like this:
`#SBATCH --gres=gpu:2`

and in my code i added this:
```
import os
os.environ["CUDA_DEVICE_ORDER"]="PCI_BUS_ID"
os.environ["CUDA_VISIBLE_DEVICES"]="0,1"
```

but when i check the number of GPUs in the training argument i always get 1:
`print("build trainer with on device:", training_args.device, "with n gpus:", training_args.n_gpu)`

the output:
`build trainer with on device: cuda:0 with n gpus: 1`

### Expected behavior

I want to use all the available GPUs

Comments:
How are you launching the python script in your bash?

this is the content of my bash script:

```
#!/bin/sh
# Options SBATCH :

#SBATCH --job-name=DSI_gpu   # Job  name
#SBATCH --mail-type=END             # Email notification 
#SBATCH --mail-user=email@esi.dz    

#SBATCH --ntasks=1                      # Number of paralel jobs
#SBATCH --cpus-per-task=4
#SBATCH --partition=GPUNodes            #  partition
#SBATCH --gres=gpu:2
#SBATCH --gres-flags=enforce-binding

# Traitement


module purge
module load singularity/3.0.3

srun singularity exec /logiciels/containerCollections/CUDA11/pytorch-NGC-21-03-py3.sif $HOME/dr_env/bin/python3.8 "path/to/python/script.py"


```

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

You're launching with python. You should use either `accelerate launch` or `torch.distributed.run` otherwise you'll get model parallel (which isn't what you're aiming for)

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Added a new without_checkpoint_model:bool param to trainer.train()
Author: stevemadere
Pull request: yes

Added a new option `without_checkpoint_model:bool` to the Trainer.train() params.

This prevents re-loading the model supplied to the constructor during a train(resume_from_checkpoint) operation so that models that the Trainer does not explicitly know how to load itself can still be resumed from a checkpoint.


Fixes #29740


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [x] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

@muellerzr , @pacman100, @amyeroberts

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_29741). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

hmmm, I'm not sure about this. This seems like a workaround that avoids addressing a problem with the trainer. 

~Could you give an example of a model that's saved out with trainer but then can't resume from checkpoint? Ideally as a small script to reproduce~ Never mind - I just saw the details in the issue :) 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Understanding loss in Training LLM
Author: mostafamdy
Pull request: no
Labels: name: Feature request

### Feature request

Hi,

I have a misunderstanding regarding training LLMs. When we train the model, we calculate the loss by having the model predict the next word and then compute the difference between the true and predicted values.

What I want to know is: when making the model predict the next words, it generates a new word based on the previously generated words. If a generated word is wrong, won't the subsequent predictions continue down the wrong path?

### Motivation

![LLMTrain-ezgif com-webp-to-jpg-converter](https://github.com/huggingface/transformers/assets/76405917/a35dd000-5c10-480f-9972-150870a43ca9)


### Your contribution

.

Comments:

No, you are mixing training and inference. Here is the explanation: 

**1. Training vs. Inference:**
   - **Training Phase:** During the training of a language model like a large language model (LLM), the model learns by predicting the next word based on the previous words in the input data. **However, it's important to understand that the model does not use its own predictions to continue the training sequence**.
   - **Inference Phase:** In contrast, during inference (or generation), the model uses the words it has just predicted to generate subsequent words. This is when the model can potentially "go down the wrong path" if it makes a wrong prediction, as each new word depends on the previously generated words.

**2. How Training Works:**
   - **Input Data:** During training, you provide the model with sequences of text from your dataset.
   - **Ground Truth:** For each sequence, the model is shown part of the sequence (e.g., the first few words) and asked to predict the next word. Importantly, whether the prediction is correct or not, the model is then given the actual next word (the "ground truth") for its next prediction. This process helps the model learn the correct associations without compounding errors from its own predictions.


### Example:

Imagine you're training a model on the phrase "The quick brown fox jumps over the lazy dog."

**Training Process:**
- **Input to Model:** "The quick brown fox jumps"
- **Correct Next Word (Ground Truth):** "over"
- **Model's Prediction:** Suppose the model predicts "around" instead of "over."
- **Feedback:** The model is informed that the correct word was "over." It does not continue with "around" for the next training step.
- **Next Input:** Regardless of the model’s incorrect prediction, the next input during training will still be "The quick brown fox jumps over" (to predict "the"), not "The quick brown fox jumps around."

This process ensures that the model is consistently trained on correct sequences, helping it learn the right patterns without the errors in its predictions affecting its training.

Thank you so much 😄❤️
Quanto serialization
Author: SunMarc
Pull request: yes

# What does this PR do ?
This PR enable serializing quanto quantized models. Needs the following [PR](https://github.com/huggingface/quanto/pull/120) in quanto. 

Draft for now

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_30049). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Fix YOLOS image processor resizing
Author: qubvel
Pull request: yes

# What does this PR do?

I observed that the YOLOS model failed with OOM while training. After some investigation, I found that square images are not properly resized by YOLOS image processor even if do_resize=True. The size parameter is not respected even if the max size of the image is larger than `longest_edge`.

Example to reproduce:
```python
import torch
from transformers import YolosImageProcessor

images = torch.randint(0, 256, (1, 3, 1500, 1500), dtype=torch.uint8)
processor = YolosImageProcessor(size={"longest_edge": 1333, "shortest_edge": 800}, do_resize=True)

# Resize the image
result = processor(images=images, return_tensors="pt")
print(result["pixel_values"].shape)

# >>> torch.Size([1, 3, 1488, 1488])
```

I am going to
1) submit a test that fails on square images
2) submit a fix

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [x] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_30436). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

@amyeroberts it's ready 🙂 

--
Finish adding support for torch.compile dynamic shapes
Author: warner-benjamin
Pull request: yes

Following #30560, this PR updates new models (Jamba & JetMoe) and DBRX to support compiling with dynamic shapes.

DBRX appears to support the FA2 SDPA kernel now, although the model is too large for me to test.

cc @ArthurZucker and @fxmarty

Comments:
Loss function defintion in the BertForSequenceClassification
Author: MostHumble
Pull request: no

Hi, I'm tryining to understand why you would give the ability to re define the loss function at each forward pass?  ie why not one time in the init?

ok nervmin, can't know labels.dtype == torch.long or
                                              labels.dtype == torch.int before runtime
https://github.com/huggingface/transformers/blob/96eb06286b63c9c93334d507e632c175d6ba8b28/src/transformers/models/bert/modeling_bert.py#L1672-L1730

Comments:
Fix typo: use_safetenstors to use_safetensors
Author: CharlesCNorton
Pull request: yes

Corrected a typo in security.md. Changed `use_safetenstors` to `use_safetensors` in the section discussing the usage of safe formats for loading models to prevent arbitrary code execution.

# What does this PR do?

This PR corrects a typo in the security.md file. The typo was found in the section discussing the usage of safe formats for loading models to prevent arbitrary code execution. The incorrect term `use_safetenstors` was corrected to `use_safetensors`.

Fixes # (issue)

## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request), Pull Request section?
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link to it if that's the case.
- [x] Did you make sure to update the documentation with your changes? Here are the [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [x] Did you write any new necessary tests?

## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag members/contributors who may be interested in your PR.

Models:

- text models: @ArthurZucker and @younesbelkada
- vision models: @amyeroberts
- speech models: @sanchit-gandhi
- graph models: @clefourrier

Library:

- flax: @sanchit-gandhi
- generate: @zucchini-nlp (visual-language models) or @gante (all others)
- pipelines: @Narsil
- tensorflow: @gante and @Rocketknight1
- tokenizers: @ArthurZucker
- trainer: @muellerzr and @SunMarc

Integrations:

- deepspeed: HF Trainer/Accelerate: @muellerzr
- ray/raytune: @richardliaw, @amogkam
- Big Model Inference: @SunMarc
- quantization (bitsandbytes, autogpt): @SunMarc and @younesbelkada

Documentation: @stevhliu and @MKhalusova

HF projects:

- accelerate: [different repo](https://github.com/huggingface/accelerate)
- datasets: [different repo](https://github.com/huggingface/datasets)
- diffusers: [different repo](https://github.com/huggingface/diffusers)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Maintained examples (not research project or legacy):

- Flax: @sanchit-gandhi
- PyTorch: See Models above and tag the person corresponding to the modality of the example.
- TensorFlow: @Rocketknight1


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_31184). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Remove copied froms for deprecated models
Author: amyeroberts
Pull request: yes

# What does this PR do?

Remove copied from statements for deprecated models.

As deprecated models are no longer tested, this prevents changes being propagated which aren't compatible with the current architecture which we would no longer catch. 

Adds a line in the utils/deprecate_models.py script to remove these lines from the files automatically when deprecating

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_31153). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
[`GemmaModel`] fix small typo
Author: ArthurZucker
Pull request: yes

# What does this PR do?

- Fixes the copy issues
- fixes the conversion

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_31202). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
How to load local code for model with `trust_remote_code=True`?
Author: LZY-the-boys
Pull request: no

### Feature request

When I use the model with `trust_remote_code=True`, I cannot directly change these remote codes because everytime I  load model it will request new codes from remote hub. So how can I avoid that ? Can I custom these codes in local? 

example:
```
model = AutoModelForSeq2SeqLM.from_pretrained('THUDM/glm-large-chinese', trust_remote_code=True)
model.forward(...) # which I want to change the code
```

### Motivation

The remote code is not always good to fit user needs. So the user should have ways to change the remote code. 

### Your contribution

if there is no other way I can sub a PR.

Comments:
Hi @LZY-the-boys, thanks for raising this issue, 

If I've understood correctly, the question being asked is how to load in a customized version of the model on the ['THUDM/glm-large-chinese' repo](https://huggingface.co/THUDM/glm-large-chinese). 

When running: 
```python
model = AutoModelForSeq2SeqLM.from_pretrained('THUDM/glm-large-chinese', trust_remote_code=True)
```

The model being downloaded will be the one defined in [THUDM/glm-large-chinese](https://huggingface.co/THUDM/glm-large-chinese). `trust_remote_code=True` is simply saying that it's OK for this model code to be downloaded and run from the hub. 

If you wish to load a local model, then this model should be saved out to either the hub or locally and the path to its location passed to `from_pretrained` e.g.:

```
model.save_pretained('path/to/my/model') # Model with adapted methods
model = ModelClass.from_pretrained('path/to/my/model', trust_remote_code=True)
```

There's more information about using models with [custom code here](https://huggingface.co/docs/transformers/v4.27.1/en/custom_models#using-a-model-with-custom-code).

OK, the `model.save_pretrained` indeed is a choice to custom the remote code in local folder, though it will copy these local files to a `transformers/local` dir and run . In early times I change the code in that temporary directory so cause the above doubt. 

@amyeroberts hi, thanks for your suggestions.

I have `save_pretrained` my checkpoints and load model `from_pretrained` using local path. However, although it load the checkpoint from my local path, it still download the model code from remote and save into `~/.cache/huggingface/modules/transformers_modules` and use it. But i need to make it use my local custom model code. How to prevent the AutoModel download remote code and use my custom code?

@dragen1860 Could you detail the code you're running to reproduce this problem and the output which is indicating that downloading is happening? 

It's possible that some files might be downloaded from the hub e.g. the config, depending wha's being called. However, calling a second time no downloads should happen provided there's no upstream changes. 

Could you try running with `HF_HUB_OFFLINE=1` set in your environment?