noamaanMulla-03/transformers-issues huggingface dataset:
Fix bug leading to missing token in GPTSanJapaneseTokenizer
Author: passaglia
Pull request: yes

# Fixes the missing comma (,) token in the GPTSanJapaneseTokenizer

The comma character "," is a token in the default vocabulary file used by GPTSanJapaneseTokenizer.

However, the vocab file parser has a bug that deletes this token and so the GPTSanJapaneseTokenizer encodes commas as the <byte 44> token instead of a unique token.

This fixes that bug.

@tanreinama @younesbelkada @ArthurZucker 

Comments:
Hey! Could you make sure to run `make fix-copies` ! Otherzise looks good to me

😅 sorry, you can remove the `copied from` to get rid of the issue with make fix copies overwriting

Remove this : `# Copied from transformers.models.gpt_neox_japanese.tokenization_gpt_neox_japanese.load_vocab_and_emoji`

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_23883). All of your documentation changes will be reflected on that endpoint.
accelerate deepspeed and gradient accumulation integrate
Author: pacman100
Pull request: yes

### What does this PR do?
1. Shift deepspeed integration to accelerate
2. Shift Gradient Accumulation to Accelerate
3. Merge after #23168 
4. no user facing change. Now user can use `accelerate launch` with trainer for DeepSpeed, e.g.:
```
accelerate launch --num_processes=2 --mixed_precision=bf16 --use_deepspeed --gradient_accumulation_steps=1 --gradient_clipping=1 --zero3_init_flag=True --zero3_save_16bit_model=False --zero_stage=3 --offload_optimizer_device=none --offload_param_device=none ./examples/pytorch/text-classification/run_glue.py  --model_name_or_path bert-base-cased   --task_name $TASK_NAME   --do_train   --do_eval   --max_seq_length 128   --per_device_train_batch_size 16   --learning_rate 5e-5   --num_train_epochs 3   --output_dir /tmp/$TASK_NAME/ --overwrite_output_dir --bf16
```

Usual run using `torchrun` and trainer args is unimpacted:
```
torchrun --nnodes 1 --nproc-per-node 2 ./examples/pytorch/text-classification/run_glue.py  --model_name_or_path bert-base-cased   --task_name $TASK_NAME   --do_train   --do_eval   --max_seq_length 128   --per_device_train_batch_size 16   --learning_rate 5e-5   --num_train_epochs 3   --output_dir /tmp/$TASK_NAME/ --overwrite_output_dir --deepspeed ~/transformers/tests/deepspeed/ds_config_zero2.json
```
5. Save and load utils are changed accordingly

Comments:
_The documentation is not available anymore as the PR was closed or merged._

> Thanks for working on this. Is the diff longer than expected because of other PRs to be merged before?

Due to updating from main, it is not showing the diff wrt previous branches. Weird. 

> Might be cool to Have Stas have a look (not pinging him here too early) once this is ready to merge and tests are confirmed to all pass.

Yes, definitely. All tests are passing already. Checked the slow tests offline.


@sgugger, now the diff is only specific to DeepSpeed changes + gradient accumulation changes + saving/loading changes wrt previous PR.

Hello @stas00, please review this PR which aims to shift the accelerate handling in Trainer to Accelerate. Thank you!
[WIP] Add internimage
Author: millionhz
Pull request: yes

# What does this PR do?

The PR adds internimage to transformers. Addresses #22240 

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

## Who can review?

Anyone in the community is free to review the PR and help with the left out work. Feel free to tag
members/contributors who may be interested in your PR.

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

@adit299 
@Weiyun1025
@amyeroberts 
@sgugger 
@stevhliu 
@MKhalusova


Comments:
All I did was copy the code from [here](https://huggingface.co/OpenGVLab/internimage_s_1k_224) as mentioned in #22240.

I couldn't figure out the test cases so I commented those out for now. 
I manually imported the model and initialized an instance to see if it works and it did. 

I will fix the failing documentation testcases. Might need some help with the technical details of the model.

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_23782). All of your documentation changes will be reflected on that endpoint.

Hi @millionhz, thanks for opening this PR and for the work adding this model! 

The linked to repo already has the [model code on the hub](https://huggingface.co/OpenGVLab/internimage_s_1k_224/blob/main/intern_image.py), and the model can be loaded directly with:

```python
from transformers import AutoModel

model = AutoModel.from_pretrained("OpenGVLab/internimage_s_1k_224", trust_remote_code=True)
``` 

so a PR to add into the transformers repo isn't necessary. 

@amyeroberts 

Oh. I opened the PR because of #22240.

I close it if its not needed.

@millionhz - no worries, it's not obvious from the issue. I'll comment on the issue and we can close both that and this PR. 

It's great that you're wanting to contribute and there's still plenty of ways you can in the library - addressing models or good first issues. We're looking forward to any future PRs :) Just remember to read the contributor's guideline and template carefully, e.g. for this PR more that 3 people were tagged.
Fix Trainer when model is loaded on a different GPU
Author: sgugger
Pull request: yes

# What does this PR do?

When a small model is loaded with `device_map="auto"` it might end up all on GPU 1, so currently `is_model_parallel` is set to `False` (cause one device) and later on the Trainer moves the model to GPU 0 which fails the execution of all the Accelerate hooks.

This PR fixes this by making sure `is_model_parallel` is set to `True` when there is one device but it's not GPU 0.

Comments:
_The documentation is not available anymore as the PR was closed or merged._
accelerate mixed precision integrate
Author: pacman100
Pull request: yes

### What does this PR do?
1. Shift Trainer's mixed precision handling to accelerate. Having smaller PR for this instead of mixing this with DDP, FSDP and DeepSpeed changes.
2. Sharded DDP and Apex are cases not supported in Accelerate and because of this I'm unable to simplify further and delete chunks of code in Trainer.
3. No user-facing changes. User can now use `accelerate launch` for launching mixed precision training in Trainer. Example given below:
```
accelerate launch --mixed_precision="bf16" run_glue.py   --model_name_or_path bert-base-cased   --task_name $TASK_NAME   --do_train   --do_eval   --max_seq_length 128   --per_device_train_batch_size 32   --learning_rate 2e-5   --num_train_epochs 3   --output_dir ~/temp/$TASK_NAME/ --overwrite_output_dir
```

the previous usage via `python` or `torchrun` is same. 
```
python run_glue.py   --model_name_or_path bert-base-cased   --task_name $TASK_NAME   --do_train   --do_eval   --max_seq_length 128   --per_device_train_batch_size 32   --learning_rate 2e-5   --num_train_epochs 3   --output_dir ~/temp/$TASK_NAME/ --fp16 --overwrite_output_dir
```

Comments:
_The documentation is not available anymore as the PR was closed or merged._
accelerate DDP integrate
Author: pacman100
Pull request: yes

### What does this PR do?
1. Move DDP preparation to Accelerate. 
2. This PR should be merged after #23148 
3. No user-facing change. Now, user can use `accelerate launch` for DDP and MP, e.g.,
```
accelerate launch --num_processes 2 --multi_gpu --mixed_precision "bf16" run_glue.py   --model_name_or_path bert-base-cased   --task_name $TASK_NAME   --do_train   --do_eval   --max_seq_length 128   --per_device_train_batch_size 16   --learning_rate 5e-5   --num_train_epochs 3   --output_dir /tmp/$TASK_NAME/ --overwrite_output_dir
```

Previous way of using torchrun works as usual:
```
torchrun --nnodes 1 --nproc-per-node 2 run_glue.py   --model_name_or_path bert-base-cased   --task_name $TASK_NAME   --do_train   --do_eval   --max_seq_length 128   --per_device_train_batch_size 16   --learning_rate 5e-5   --num_train_epochs 3   --output_dir /tmp/$TASK_NAME/ --overwrite_output_dir --bf16
```

Empirical nuances that I noticed:
1. As DDP uses Accelerate, the LR Scheduler is run `num_processes` per step. Previously, it was only run once per step. Because of this, lr decreases rapidly when using Accelerate's integration. In the above example, I had to increase LR from 2e-5 to 5e-5 to account for this behaviour in order to maintain the performance.

Comments:
_The documentation is not available anymore as the PR was closed or merged._
[`core`] Add no split modules support for all models
Author: younesbelkada
Pull request: yes

# What does this PR do?

Fixes: #23816 and other related issues

## Motivation

With the increase of the usage of code on the Hub features and more and more models pushed on that direction, I am convinced that it would be great to easily support accelerate loading of these models out of the box, without having to open PRs on the Hub and wait for authors approvals on each repo (as the changes needs to be duplicated across all repos, which can be redundant).

## Description

This PR addresses this, and adds support of `no_split_modules` on the `from_pretrained` method. Allowing superusers to load models using accelerate (for loading models in 8bit / 4bit for example), for models where the code is hosted on the Hub.

## To reproduce:

```python
from transformers import AutoModelForCausalLM, AutoTokenizer

model_name = 'mosaicml/mpt-7b-instruct'

tokenizer = AutoTokenizer.from_pretrained("EleutherAI/gpt-neox-20b")
model = AutoModelForCausalLM.from_pretrained(
    model_name, 
    load_in_8bit=True,
    device_map="auto",
    no_split_modules=["MPTBlock"],
    trust_remote_code=True,
)

prompt = "What is the boiling point of Nitrogen?"

input_ids = tokenizer(prompt, return_tensors="pt").input_ids.to(0)
out = model.generate(input_ids)
print(tokenizer.decode(out[0], skip_special_tokens=True)) 
>>> What is the boiling point of Nitrogen?\n The boiling point of Nitrogen is 77.37
```

cc @ArthurZucker @sgugger 

Once the PoC approved, will add a section on the docs

Comments:
_The documentation is not available anymore as the PR was closed or merged._

I don't think this is a good idea and I don't want `from_pretrained` to end up with 96 arguments. It's something that's very internal to the model, so should definitely be in the model code. If a user opens a PR they can directly used the updated code by specifying a revision argument.

Sounds good to me! Closing the PR
Avoid saving tied weights with sharded checkpoints
Author: nmntz
Pull request: no

It seems that when sharding a checkpoint we untie weights which makes them take more space
```python
import torch
from transformers import GPT2LMHeadModel, GPT2Config

config = GPT2Config()
model = GPT2LMHeadModel(config)
assert id(model.transformer.wte.weight) == id(model.lm_head.weight)
model.save_pretrained("gpt2-tied-weights")

config.tie_word_embeddings = False
model = GPT2LMHeadModel(config)
assert id(model.transformer.wte.weight) != id(model.lm_head.weight)
model.save_pretrained("gpt2-untied-weights")

config = GPT2Config()
model = GPT2LMHeadModel(config)
assert id(model.transformer.wte.weight) == id(model.lm_head.weight)
model.save_pretrained("gpt2-tied-weights-sharded", max_shard_size="100MB")

config.tie_word_embeddings = False
model = GPT2LMHeadModel(config)
assert id(model.transformer.wte.weight) != id(model.lm_head.weight)
model.save_pretrained("gpt2-untied-weights-sharded", max_shard_size="100MB")
```

When checking the space taken by these checkpoints:
```
$ du -sh gpt2*
475M    gpt2-tied-weights
622M    gpt2-tied-weights-sharded   # MUST BE 475M
622M    gpt2-untied-weights
622M    gpt2-untied-weights-sharded
```
cc @ArthurZucker @thomasw21 

Comments:
I don't think that developing the logic that would avoid this is really worth the time it will require, but we can leave this issue as a reference point.

Hum I don't think it should take that long. I can have a go at it. I do think it's an easy win.
Same size on memory usage when loading gpt2 from torch.float16 and 8bit Quantization
Author: czhang-trinity
Pull request: no

### System Info

- `transformers` version: 4.30.0.dev0
- Platform: Linux-5.15.0-1026-aws-x86_64-with-glibc2.29
- Python version: 3.8.10
- Huggingface_hub version: 0.14.1
- Safetensors version: 0.3.1
- PyTorch version (GPU?): 2.0.1+cu117 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?:No

### Who can help?

@pacman100 

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

```
from transformers import AutoModelForCausalLM
from transformers import BitsAndBytesConfig
import torch

model_8bit = AutoModelForCausalLM.from_pretrained(
    "gpt2", 
    load_in_8bit=True,
    device_map='auto',
)
print('8 bit: ', model_8bit.get_memory_footprint())


model_float16 = AutoModelForCausalLM.from_pretrained(
    "gpt2", 
    load_in_8bit=False, 
    device_map='auto',
    torch_dtype=torch.float16
)
print('float16: ', model_float16.get_memory_footprint())
```

### Expected behavior

Result is
 8 bit:  274045464
float16:  261462552

Should see 8bit smaller than float16

Comments:
cc @younesbelkada 

Hi @czhang-trinity 
Thanks for the issue, running your script on the current main branch gives me:

```bash 
8 bit:  261462552
16 bit:  261462552
```

This is because GPT2 uses Conv1D in replacement to all linear layers. Therefore the 8bit conversion ends up converting no Linear layers !
```bash
GPT2LMHeadModel(
  (transformer): GPT2Model(
    (wte): Embedding(50257, 768)
    (wpe): Embedding(1024, 768)
    (drop): Dropout(p=0.1, inplace=False)
    (h): ModuleList(
      (0-11): 12 x GPT2Block(
        (ln_1): LayerNorm((768,), eps=1e-05, elementwise_affine=True)
        (attn): GPT2Attention(
          (c_attn): Conv1D()
          (c_proj): Conv1D()
          (attn_dropout): Dropout(p=0.1, inplace=False)
          (resid_dropout): Dropout(p=0.1, inplace=False)
        )
        (ln_2): LayerNorm((768,), eps=1e-05, elementwise_affine=True)
        (mlp): GPT2MLP(
          (c_fc): Conv1D()
          (c_proj): Conv1D()
          (act): NewGELUActivation()
          (dropout): Dropout(p=0.1, inplace=False)
        )
      )
    )
    (ln_f): LayerNorm((768,), eps=1e-05, elementwise_affine=True)
  )
  (lm_head): Linear(in_features=768, out_features=50257, bias=False)
)
```
[`bnb`] add warning when no linear 
Author: younesbelkada
Pull request: yes

# What does this PR do?

This PR adds a new warning for users in case `replace_with_bnb_linear` hasn't replace any module, this would avoid any confusion for users

Fixes https://github.com/huggingface/transformers/issues/23807

cc @sgugger 

Comments:
_The documentation is not available anymore as the PR was closed or merged._
fix: Replace `add_prefix_space` in `get_prompt_ids` with manual space for FastTokenizer compatibility
Author: connor-henderson
Pull request: yes

# What does this PR do?
Fixes #23764 

As discussed in the issue the 'FastTokenizer' for Whisper and other models does not accept `add_prefix_space` as an argument to tokenize, so to make `get_prompt_ids` compatible across both slow and fast tokenizers this was replaced with `" " + text`

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [x] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [x] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- text models: @ArthurZucker and @younesbelkada
- vision models: @amyeroberts
- speech models: @sanchit-gandhi
- graph models: @clefourrier

Library:

- flax: @sanchit-gandhi
- generate: @gante
- pipelines: @Narsil
- tensorflow: @gante and @Rocketknight1
- tokenizers: @ArthurZucker
- trainer: @sgugger

Integrations:

- deepspeed: HF Trainer: @stas00, Accelerate: @pacman100
- ray/raytune: @richardliaw, @amogkam

Documentation: @sgugger, @stevhliu and @MKhalusova

HF projects:

- accelerate: [different repo](https://github.com/huggingface/accelerate)
- datasets: [different repo](https://github.com/huggingface/datasets)
- diffusers: [different repo](https://github.com/huggingface/diffusers)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Maintained examples (not research project or legacy):

- Flax: @sanchit-gandhi
- PyTorch: @sgugger
- TensorFlow: @Rocketknight1

 -->
@hollance @sanchit-gandhi 

Comments:
_The documentation is not available anymore as the PR was closed or merged._

@sanchit-gandhi for sure just pushed something up
Move import check to before state reset
Author: muellerzr
Pull request: yes
Labels: name: bug

# What does this PR do?

This PR moves the reset of the `AcceleratorState` to be under the check for if it is available, and guards the resetting better.

Fixes # (issue)

Fixes https://github.com/huggingface/transformers/issues/23898


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

@sgugger 

Comments:
_The documentation is not available anymore as the PR was closed or merged._

cc @pacman100 so it's on your radar too that we can do this :) 

How about adding accelerate as a required dependency in the setup.py? 


torch is not a hard dependency of Transformers.

Which is why it's part of the `[torch]` reqs :)
Support shared tensors
Author: thomasw21
Pull request: yes

# What does this PR do?

- Fixes #23868

We can uniquely hash the storage by computing `data_ptr()` and `nbytes()` since storages are 1D contiguous buffers. We use that to find tensors that share the same storage. And if we do find them, we put those within the same "block" relying on underlying code to optimize serialization.


Comments:
_The documentation is not available anymore as the PR was closed or merged._

Failing test seems unrelated?

Nice 🔥 

Seems like this requires the latest version of safetensors otherwise getting 
```python 
E   RuntimeError: Failed to import transformers.modeling_utils because of the following error (look up to see its traceback):
E   cannot import name 'storage_ptr' from 'safetensors.torch' (/opt/conda/envs/py39/lib/python3.9/site-packages/safetensors/torch.py)
```
should probably update the setup? 

Yes @muellerzr made a PR: #23911 

Woops! Thank you @muellerzr !
Upgrade safetensors version
Author: muellerzr
Pull request: yes

# What does this PR do?

Upgrades `safetensors` version in the requirements to avoid the following error message:
(I had 0.3.0)

```python
src/transformers/models/gpt2/modeling_gpt2.py:38: in <module>
    from ...modeling_utils import PreTrainedModel, SequenceSummary
src/transformers/modeling_utils.py:40: in <module>
    from .pytorch_utils import (  # noqa: F401
src/transformers/pytorch_utils.py:19: in <module>
    from safetensors.torch import storage_ptr, storage_size
E   ImportError: cannot import name 'storage_ptr' from 'safetensors.torch' (/home/zach_mueller_huggingface_co/miniconda3/envs/accelerate/lib/python3.9/site-packages/safetensors/torch.py)
```

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

@sgugger 

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_23911). All of your documentation changes will be reflected on that endpoint.