noamaanMulla-03/transformers-issues huggingface dataset:
Fix sagemaker DP/MP
Author: muellerzr
Pull request: yes

# What does this PR do?

Fixes the broken sagemaker tests, proved it works. 

Solves https://github.com/huggingface/transformers/issues/23390

Needs to be coordinated with the Accelerate pr as well

Comments:
_The documentation is not available anymore as the PR was closed or merged._

Hello, devs. I encountered this error on Google Colab (Python 3.10) and transformers 4.30.1.

```
❱ 3386 │   │   │   self.args.distributed_state is None and self.local_rank ! 

AttributeError: 'CLTrainer' object has no attribute 'local_rank'
```

It looks like the line was added by this change.

By looking through the code, I suspect `self.local_rank` should be `self.args.local_rank`. I'm fairly new to this library, so apologies if my guess is wrong.

Indeed. Would you like to open a PR with the fix?

Sure, I can open a PR in a few days. But I'm actually pretty new to this repo, so please feel free to make a quick fix for that.

cc @muellerzr might be worth a quick fix.

Hi. I just opened #24297.
Split common test from core tests
Author: sgugger
Pull request: yes

# What does this PR do?

This PR aims at cleaning the tests files like `test_modeling_common.py` which contain two distinct things: the common tester which is reused by all model tests, but also some core tests of `modeling_utils.py`. This PR split this file (and all similar) into 2: the common tester mixin stays in `test_modeling_common.py` and all other tests go to `test_modeling_utils.py`.

Comments:
_The documentation is not available anymore as the PR was closed or merged._

@ydshieh It's just the addition of new tests for modeling_utils or tokenization_utils which is really painful since those files are ~4k lines.
__init__() missing 1 required positional argument
Author: Changgeng-Wei
Pull request: no

### System Info

- `transformers` version: 4.9.2
- Platform: Windows-10-10.0.19041-SP0
- Python version: 3.7.11
- PyTorch version (GPU?): 1.9.0+cpu (False)
- Tensorflow version (GPU?): 2.6.0 (False)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: No
- Using distributed or parallel set-up in script?:No


### Who can help?

@you

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

from transformers.modeling_utils import PretrainedConfig

class TClass(PretrainedConfig):
    def __init__(self, config):
        super(TClass, self).__init__()
        self.config = config


if __name__ == '__main__':
    c = TClass(config='setting')
    print(c)

### Expected behavior

Hi!
I'am a green hands, I wanted to define a class which is based on PretrainedConfig and sent a variate when initiate the object, but I faced a issue discribed by beflowing text. I change many transformers versions and python versions, but the issue is still happened, could you help me to solve it? Thanks a lot!
 File "F:/workProject//test/test.py", line 12, in <module>
    print(c)
  File "E:\ProgramData\Anaconda3\envs\py37_tf250_torch\lib\site-packages\transformers\configuration_utils.py", line 613, in __repr__
    return f"{self.__class__.__name__} {self.to_json_string()}"
  File "E:\ProgramData\Anaconda3\envs\py37_tf250_torch\lib\site-packages\transformers\configuration_utils.py", line 674, in to_json_string
    config_dict = self.to_diff_dict()
  File "E:\ProgramData\Anaconda3\envs\py37_tf250_torch\lib\site-packages\transformers\configuration_utils.py", line 629, in to_diff_dict
    class_config_dict = self.__class__().to_dict() if not self.is_composition else {}
TypeError: __init__() missing 1 required positional argument: 'config'"


Comments:
Hi there! There is no way we will be able to help you without seeing the code you are running.

> The code is below

from transformers.modeling_utils import PretrainedConfig

class TClass(PretrainedConfig):
def init(self, config):
super(TClass, self).init()
self.config = config

if name == 'main':
c = TClass(config='setting')
print(c)

Hi! I'm having the same problem... is there a documentation on how to extend transformers configs?
Loading fp16 model as fp32 when using .from_retrained()
Author: EdanZizo
Pull request: no

### System Info


When loading GPTJ from GPTJForCausalLM.from_pretrained() loading 16bit model which should be approximately 12GB, it instead has a model size ~23GB which is the full 32 bit weights.

the code:
```
device = torch.device('cuda:0' if torch.cuda.is_available() else 'cpu')

tokenizer = AutoTokenizer.from_pretrained("ThisIsMyUsername69/gpt-j-6B-16bit")
config = AutoConfig.from_pretrained("ThisIsMyUsername69/gpt-j-6B-16bit", torch_dtype=torch.float16)
model = GPTJForCausalLM.from_pretrained("ThisIsMyUsername69/gpt-j-6B-16bit", config=config)
```
I've tried multiple ways of trying to load in 16 bit, from_config, with or without autoconfig, regardless of everything it seems to always use 23GB of VRAM except with EleutherAI/gpt-j-6B using revision float16.

The model has memory footprint of 23194MiB.  ="ThisIsMyUsername69/gpt-j-6B-16bit" and "nlpcloud/instruct-gpt-j-fp16" both give the larger model size.

I have tried giving the parameter ' revision="float16" ' and it gives the same response.  

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

1. run the code above
2. observe model size

### Expected behavior

The model size should be approximately 11GB however it is giving the full model weight (32 bit float) size.

Comments:
cc @younesbelkada 

Hi @EdanZizo 
to load the model with the desired dtype directly from the config I believe you should use `torch_dtype="auto"` in `GPTJForCausalLM.from_pretrained("ThisIsMyUsername69/gpt-j-6B-16bit", config=config)`. But note that the canonical way to load any model in half precision is: 
```python
GPTJForCausalLM.from_pretrained("ThisIsMyUsername69/gpt-j-6B-16bit", torch_dtype=torch.float16)
```

It is working now, thanks for your help.
Fix image segmentation tool bug
Author: amyeroberts
Pull request: yes

# What does this PR do?

Currently tools using the ImageSegmentation tool fail because the parameters for the image processor are overridden with the input image dimensions. This results in incompatible input dimensions being passed to the model.

This PR removes this logic in the `encode` method and removes the resizing in the tests which only happened for the segmentation tool and hid the issue. 

Fixes #23328 


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [x] Did you write any new necessary tests?


Comments:
@sgugger @LysandreJik The fix is pretty simple, but in the spirit of Chesterton's fence, I wasn't sure/couldn't remember the reason for the `self.pre_processor.image_processor.size = ...` logic so wanted to ask you both in case this is breaking assumptions elsewhere. 

_The documentation is not available anymore as the PR was closed or merged._

This seems reasonable! I had run into issues where I couldn't segment images with different sizes when implementing it.

I have tried with different flavors of the following 

![image](https://github.com/huggingface/transformers/assets/30755778/0e895dc8-c4ce-41fe-a864-4d298571d4ac)

and it seems to work well even without specifying the size.

Your change looks good to me @amyeroberts; are you aware of sizes that may not work with this model?

@LysandreJik It *should* all be OK if the image processor matches the model: there shouldn't be any sizes that don't work because the image processor will resize as needed. 

In terms of sizes that won't work:
* We can't input images with either height or width smaller that the model's patch size - 16 by default
* Images which (image_height // patch_size) * (image_width // patch_size) > max_sequence_length
* Non-square images where `(image_height // patch_size) != (image_width // patch_size)`. Just looking high-level at the model, I think it could be reworked to accept non-square images but haven't dug into it deeply.
[Docs] Improve docs for MMS loading of other languages
Author: patrickvonplaten
Pull request: yes

# What does this PR do?

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

Clarifies #24223 and improves docs of MMS.



Comments:
_The documentation is not available anymore as the PR was closed or merged._
[Docs] Fix the paper URL for MMS model
Author: hitchhicker
Pull request: yes

# What does this PR do?

Fixes the paper link for the mms model, the wrong URL is for the paper `XLS-R: SELF-SUPERVISED CROSS-LINGUAL SPEECH REPRESENTATION LEARNING AT SCALE` not `Scaling Speech Technology to 1,000+ Languages`.

Comments:
_The documentation is not available anymore as the PR was closed or merged._

Thanks for the quick review! @amyeroberts Would you please merge it?
torch.jit support
Author: chuckhope
Pull request: no

### Feature request

Hi there, is torch.jit used by default for model inference in the transformers library, for example, in the Auto series APIs? If not, why isn't it used by default? Thank you.

### Motivation

None

### Your contribution

None

Comments:
No it's not used by default since the Transformers library can't know whether you are going to use your model for training or inference. It also comes with constraints (like duplicating shared weights) so it's up to the user to activate it if the situation suits their needs. You can pass `torchscript=True` when loading your model to have the jit-compilation done for you.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Update language_modeling.py
Author: rishabhstha
Pull request: yes

Title: Add truncate_seq_pair function to TextDatasetForNextSentencePrediction

Description:
This PR adds a `truncate_seq_pair()` function to the `TextDatasetForNextSentencePrediction` class, inside the `create_examples_from_document()` function. This function truncates the sequences if they exceed the maximum number of tokens, which is useful when dealing with very long input texts. By including this function, the generated examples will adhere to the maximum sequence length constraint, which is important for the proper functioning of the model during pre-training.

Fixes # (issue)

## Before submitting
- [x] I read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section.
- [x] I wrote any new necessary tests.

## Who can review?
- text models: @ArthurZucker and @younesbelkada
- tokenizers: @ArthurZucker
- trainer: @sgugger

Please let me know if there are any changes or improvements needed.

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_23200). All of your documentation changes will be reflected on that endpoint.

This code is deprecated and not maintained anymore. To preprocess your data, we recommend you use the `datasets` library.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Update tokenizer_summary.mdx (grammar)
Author: belladoreai
Pull request: yes

# What does this PR do?

Update docs with minor grammar fix

## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [x] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

Comments:
_The documentation is not available anymore as the PR was closed or merged._
Make `can_generate` as class method
Author: ydshieh
Pull request: yes

# What does this PR do?

Make `can_generate` as class method, so we can check a model (class) can generate or not without loading/creating a model instance.

(The goal of this PR is not to address the issue regarding how to check `is_encoder`, `is_decoder` etc. discussed offline).

Comments:
_The documentation is not available anymore as the PR was closed or merged._
FSDP cuda out of memory during checkpoint saving
Author: li-plus
Pull request: no

### System Info

- `transformers` version: 4.30.0.dev0
- Platform: Linux-4.14.81.bm.22-amd64-x86_64-with-glibc2.24
- Python version: 3.9.16
- Huggingface_hub version: 0.14.1
- Safetensors version: not installed
- PyTorch version (GPU?): 2.0.1+cu117 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: yes
- Using distributed or parallel set-up in script?: yes


### Who can help?

@stas00 @pacman100 

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

I'm running https://github.com/tatsu-lab/stanford_alpaca/blob/main/train.py on 32GB GPUs to finetune LLaMA-7B with FSDP turned on. The training process goes well but I get cuda oom when saving the final checkpoint. The entire fp32 state dict cannot fit in my gpu memory. A possible solution is to offload state dict to cpu mentioned in https://github.com/pytorch/pytorch/issues/98823? Is there any better way to handle this?

```
/data00/home/lijiahao.plus/miniconda3/envs/mlir/lib/python3.9/site-packages/torch/distributed/fsdp/_state_dict_utils.py:312: UserWarning: Failed to clone() tensor with name _fsdp_wrapped_module.model.layers.31.mlp.gate_proj.weight on rank 2. This may mean that this state_dict entry could point to invalid memory regions after returning from state_dict() call if this parameter is managed by FSDP. Please check clone implementation of _fsdp_wrapped_module.model.layers.31.mlp.gate_proj.weight. Error: CUDA out of memory. Tried to allocate 172.00 MiB (GPU 2; 31.75 GiB total capacity; 29.54 GiB already allocated; 39.75 MiB free; 30.38 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation.  See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF
```


### Expected behavior

Not oom on saving checkpoints.

Comments:
Not sure why you tagged myself or Sourab - we have worked on Deepspeed integration, and you're welcome to ask questions if you use that. I personally don't know anything about FSDP - Deepspeed works perfectly well and FSDP implements the same ZeRO protocol that Deepspeed innovated. 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Beam search type
Author: jprivera44
Pull request: yes

# What does this PR do?




This PR fixes issue #22856 , which has a type mismatch between the returned value, and the type hinting in the  BeamSeachScorer, process function. Previously the type hint was set to a Tuple which was inconsistent with the returned Dict value. I've changed them to be consistent and ran the following test cases.

1. Performed print statements of the process annotations.
- Before changes return  type was typing.Tuple[torch.Tensor]}
- After changes return type was typing.Dict[str, torch.Tensor]

2. Ran PyTest for the test_beam_search.py with all 6 test cases passing. 


# Motivation and Context

To ensure high quality code within the HuggingFace repo.

## Who can review?
@gante 




Comments:
_The documentation is not available anymore as the PR was closed or merged._

Absolutely :)
[`SwitchTransformers`] Fix return values
Author: ArthurZucker
Pull request: yes

# What does this PR do?
 
The previous version of the code would always return `None` values for the router losses, since the `if output_router_probs` was wrapper around `if labels is not None`. But router logits can still be computed wthout  labels. 
Also returns tensors instead of None, this follows our usual API, and is less prone to errors. 

Comments:
_The documentation is not available anymore as the PR was closed or merged._
Fix functional TF Whisper and modernize tests
Author: Rocketknight1
Pull request: yes

There was a regression in 4.30 that affects functional construction of Whisper models in certain cases, my bad!

In an attempt to avoid this in future, I modified the `test_compile_tf_model` test. These tests were quite old and weren't that relevant for how we do things now, and were also quite slow. I pared the test down to the actual thing we want to test (functional construction with `tf.keras.Input` and potentially-unknown shape dimensions), which should make it fast enough to run in the live CI, as well as giving us more useful info about regressions like this in future.

Fixes #24291 

Comments:
_The documentation is not available anymore as the PR was closed or merged._

Update: The test is still very slow in some cases in the CI. I'm going to mark it as `@slow` before merging this PR after all, but I'll leave it as-is for now so I can fix any issues it raises before merging.

Quick note: I marked the test as slow (it was slow before too). I ran everything locally and all models passed, so hopefully it should still look good on the nightly CI after this is merged.

cc @amyeroberts for core maintainer review and we should be good to go! (Also sorry Amy if I'm spamming you with lots of PRs this week)

Thanks for the ping re-regression. This is way too big of a PR to be included in a patch though so I suggest making a separate small PR for the part that would need to go in a patch, or decide this won't go in a patch and tell users to wait for 4.31.

Ah, sorry! I didn't mean to imply there'd be a patch release - this doesn't affect too many people (only a very specific subset of Whisper users who are exporting models using the functional API), so it should be fine to wait until 4.31. I can just tell affected people to install from `main` until then.
Cannot serialize Whisper decoder layer in a keras model
Author: perretv
Pull request: no

### System Info

- `transformers` version: 4.30.2
- Platform: Linux-4.19.0-24-cloud-amd64-x86_64-with-glibc2.28
- Python version: 3.11.3
- Huggingface_hub version: 0.15.1
- Safetensors version: 0.3.1
- PyTorch version (GPU?): not installed (NA)
- Tensorflow version (GPU?): 2.12.0 (True)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: yes
- Using distributed or parallel set-up in script?: no

### Who can help?

@Rocketknight1

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [x] My own task or dataset (give details below)

### Reproduction

Changes introduced in #23760 and released in transformers [v4.30.0](https://github.com/huggingface/transformers/releases/tag/v4.30.0) are breaking the ability to serialize a keras model that contains a Whisper decoder layer.
Here is a minimal reproducible example:
```python
from transformers import TFWhisperModel
import tensorflow as tf

whisper = TFWhisperModel.from_pretrained("openai/whisper-tiny")
inp = tf.keras.Input((80, 3000))
stack = whisper.get_encoder()(inp)
decoder_input_ids = tf.ones((tf.shape(inp)[0], 1), dtype=tf.int32)* whisper.config.decoder_start_token_id
stack = whisper.get_decoder()(input_ids=decoder_input_ids, encoder_hidden_states=stack.last_hidden_state)
model = tf.keras.Model(inp, stack)
model.summary()
model.save("whisper-tiny-custom")
```

With `transformers>=4.30.0`, this minimal reproducible example will raise the error:
```
OperatorNotAllowedInGraphError: Exception encountered when calling layer 'decoder' (type TFWhisperDecoder).

Using a symbolic `tf.Tensor` as a Python `bool` is not allowed: AutoGraph did convert this function. This might indicate you are trying to use an unsupported feature.

Call arguments received by layer 'decoder' (type TFWhisperDecoder):
  • self=tf.Tensor(shape=(1, 1), dtype=int32)
  • input_ids=None
  • attention_mask=None
  • position_ids=None
  • encoder_hidden_states=tf.Tensor(shape=(None, 1500, 384), dtype=float32)
  • head_mask=None
  • cross_attn_head_mask=None
  • past_key_values=None
  • inputs_embeds=None
  • use_cache=None
  • output_attentions=None
  • output_hidden_states=None
  • return_dict=None
  • training=True
```

### Expected behavior

Keras model serialization should succeed.

Comments:
Hi @perretv yes, this looks like a regression. Investigating now, hopefully we can make a quick patch!

Hi @perretv, the fix has now been merged. You can `pip install git+https://github.com/huggingface/transformers.git` to install from `main` and use it immediately. It'll be included in the next 4.31 release of `transformers`, after which you can go back to normal pip installs.