noamaanMulla-03/transformers-issues huggingface dataset:
DataCollatorForLanguageModeling ignores manual -100 labels for loss masking
Author: kabachuha
Pull request: no

### System Info

transformers version https://github.com/huggingface/transformers/commit/5936c8c57ccb2bda3b3f28856a7ef992c5c9f451
Kubuntu 22.04 LTS
NVIDIA 4090

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

Info from https://github.com/oobabooga/text-generation-webui/issues/4031

Consider oobabooga's training script

https://github.com/oobabooga/text-generation-webui/blob/ee7bf49804bfb53775424d4c12294445497b610f/modules/training.py#L341-L358

It uses -100 labels in attempt to mask the loss for the parts we don't need to train or, like when we use `## RESPONSE` masking for better training of Alpaca lora, excluding retraining on the beginning part of the dialogue.

---

To reproduce just put prints to DataCollatorForLanguageModeling's `torch_call` function:

```python
labels = batch["input_ids"].clone()
print('INPUT LABELS')
print(labels)
if self.tokenizer.pad_token_id is not None:
    labels[labels == self.tokenizer.pad_token_id] = -100
print('OVERWRITTEN LABELS')
batch["labels"] = labels
print(batch["labels"])
```

### Expected behavior

The collator should not overwrite the manual set masks for loss calculation, as it creates confusion when the trainers set the labels to -100 only to find out it doesn't actually mask the loss. This is extremely non-obvious (need to look up the very base transformers code) and has already cost me (and countless clueless users) days of wasted high-end GPU time

Comments:
Hey @kabachuha, this seems like a reasonable feature request! Would you like to open a PR to offer your fix?

Thanks for your interest! Sure, I can, it is quite short.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
[`core` / `Quantization` ] AWQ integration
Author: younesbelkada
Pull request: yes

# What does this PR do?

As per title, this PR adds the AWQ inference support in transformers. 

- Original paper: https://arxiv.org/abs/2306.00978
- Original LLM-AWQ repository: https://github.com/mit-han-lab/llm-awq/ 
- Auto-AWQ repository: https://github.com/casper-hansen/AutoAWQ

![Screenshot 2023-10-24 at 17 56 56](https://github.com/huggingface/transformers/assets/49240599/9c956933-f1fa-45b1-904e-884249074cd0)

AWQ is a new and popular quantization scheme, already used in various libraries such as TGI, vllm, etc. and known to be faster than GPTQ models according to some benchmarks

Contrary to GPTQ, in this integration we want to support **inference only** - since the ecosystem is quite mature with respect to quantizing a model, we will publicize different routes for that purpose, such as using `auto-awq`, the original repository or optimum Neural Compressor.

For now I have pushed a 'test' model under this repository: https://huggingface.co/ybelkada/test-mistral-7b-v0.1-awq but we plan to support all AWQ weights from TheBloke. For running experiments using this PR, you can first `pip install autoawq` then run:

<details> 

```python
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "ybelkada/test-mistral-7b-v0.1-awq"

tok = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, torch_dtype=torch.float16, low_cpu_mem_usage=True).to(0)

print(model)

text = ["Hello my name is", "hi"]
input_ids = tok.encode(text, return_tensors="pt").to(0)

output = model.generate(input_ids, max_new_tokens=40)
print(tok.batch_decode(output, skip_special_tokens=True))
```

</details> 

## TODO:

- [x] Benchmarks
- [x] Documentation
- [ ] Support fused modules
- [x] Colab inference demo
- [x] Write tests
- [ ] Support weights that have been quantized with optimum NC
- [x] Support weights that have been quantized with llm-awq

cc @fxmarty @SunMarc @casper-hansen @TheBloke @IlyasMoutawwakil 

Comments:
_The documentation is not available anymore as the PR was closed or merged._

This is super exciting to see! The original repository does not support TheBloke’s quants, they were made with AutoAWQ - perhaps an argument to route to AutoAWQ for compatibility. 

Thanks @amyeroberts for your review! I have replied to your questions above, I believe that all the points are already being addressed. Let me know if I missed anything

Thanks for all your reviews @amyeroberts @ArthurZucker @SunMarc @casper-hansen !
Translating `en/main_classes` folder docs to Japanese 🇯🇵
Author: rajveer43
Pull request: yes

# What does this PR do?

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

Fixes #26893

@stevhliu 


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_26894). All of your documentation changes will be reflected on that endpoint.

see @stevhliu all checks passed

I restructured toctree.yml again by following english tochtree have alook 

@stevhliu Other PR is ready for `en/tasks` files and currently working on `model_doc` as well. as `model_doc` has too many files in it, what should be the best way to approach `model_doc` files?

I would approach it by breaking the contents of `model_doc` into smaller chunks which makes it easier to translate and review. Maybe translate ~10 models in each PR?

> I would approach it by breaking the contents of `model_doc` into smaller chunks which makes it easier to translate and review. Maybe translate ~10 models in each PR?

@stevhliu Sounds good! I will work on 10-12 files in each PR review them and add..
Fix docstring get maskformer resize output image size
Author: wesleylp
Pull request: yes

# What does this PR do?
Fix `get_maskformer_resize_output_image_size docstring`

## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?
@amyeroberts 

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_27196). All of your documentation changes will be reflected on that endpoint.
fixing docstring in get_resize_output_image_size function
Author: wesleylp
Pull request: yes

# What does this PR do?
Fixes #27185 



## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?
@amyeroberts 
@rafaelpadilla 




Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_27191). All of your documentation changes will be reflected on that endpoint.
Small issue in docstring found
Author: rafaelpadilla
Pull request: no
Labels: name: Good First Issue

function `get_resize_output_image_size(...)` is being `# Copied from` in many models [code](https://github.com/search?q=repo%3Ahuggingface%2Ftransformers+get_resize_output_image_size&type=code). 

However, the docstring has some issues like the `input_image` argument is missing and `image_size` is in docstring but never used.

Example found in [image_processing_deformable_detr.py](https://github.com/huggingface/transformers/blob/6b7f8ff1f3db0b21afbedb322770c292bc8dedae/src/transformers/models/deformable_detr/image_processing_deformable_detr.py#L126)

```python
def get_resize_output_image_size(
    input_image: np.ndarray,
    size: Union[int, Tuple[int, int], List[int]],
    max_size: Optional[int] = None,
    input_data_format: Optional[Union[str, ChannelDimension]] = None,
) -> Tuple[int, int]:
    """
    Computes the output image size given the input image size and the desired output size. If the desired output size
    is a tuple or list, the output image size is returned as is. If the desired output size is an integer, the output
    image size is computed by keeping the aspect ratio of the input image size.

    Args:
        image_size (`Tuple[int, int]`):
            The input image size.
        size (`int`):
            The desired output size.
        max_size (`int`, *optional*):
            The maximum allowed output size.
        input_data_format (`ChannelDimension` or `str`, *optional*):
            The channel dimension format of the input image. If not provided, it will be inferred from the input image.
    """
    .....
    
```

Comments:
@rafaelpadilla can I work on that?
CLIP Typo
Author: fabfish
Pull request: no

Hi, I just found a typo in the CLIP model at

https://github.com/huggingface/transformers/blob/50378cbf6c1fd8717a74b36c352f57f9a73e7282/src/transformers/models/clip/modeling_clip.py#L817C26-L817C26

where the "layernorm" is misspelled as "layrnorm" in "pre_layrnorm" but correct in "post_layernorm", guess it would be better to fix it in the code and pretrained weights to keep consistency.

Comments:
I would like to fix it. Please assign me

Hi @fabfish, thanks for reporting! 

Unfortunately, we can't update the names of layers and modules in a model once its weights have been released, as this would prevent them from being properly loaded into the model after the name change. 

> Hi @fabfish, thanks for reporting!
> 
> Unfortunately, we can't update the names of layers and modules in a model once its weights have been released, as this would prevent them from being properly loaded into the model after the name change.

That's unfortunate, but I get it. Thank you for your response!
Added cache_block_outputs option to enable GPTQ for non-regular models
Author: AlexKoff88
Pull request: yes

cache_block_outputs enables the collection of the block output to speed up GPTQ process. However, it does not work for some models such as ChatGLM where the LayerNorm is the first layer in the block.
Just compare:

OPT structure:
model.decoder.layers.0.self_attn
model.decoder.layers.0.self_attn.k_proj
model.decoder.layers.0.self_attn.v_proj
model.decoder.layers.0.self_attn.q_proj
model.decoder.layers.0.self_attn.out_proj
model.decoder.layers.0.activation_fn
model.decoder.layers.0.self_attn_layer_norm
model.decoder.layers.0.fc1
model.decoder.layers.0.fc2
model.decoder.layers.0.final_layer_norm

ChatGLM structure:
transformer.encoder.layers.0
transformer.encoder.layers.0.input_layernorm
transformer.encoder.layers.0.self_attention
transformer.encoder.layers.0.self_attention.query_key_value
transformer.encoder.layers.0.self_attention.core_attention
transformer.encoder.layers.0.self_attention.core_attention.attention_dropout
transformer.encoder.layers.0.self_attention.dense
transformer.encoder.layers.0.post_attention_layernorm
transformer.encoder.layers.0.mlp
transformer.encoder.layers.0.mlp.dense_h_to_4h
transformer.encoder.layers.0.mlp.dense_4h_to_h

The solution is to disable SA block output caching and collect the quantizing block inputs starting from the beginning of the model. It slows down the optimization a bit but works more stable.

Related PR to Optimum: https://github.com/huggingface/optimum/pull/1479

Comments:
@AlexKoff88 please run `make style` to fix the tests. 

> @AlexKoff88 please run `make style` to fix the tests.

Fixed

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_27032). All of your documentation changes will be reflected on that endpoint.
chore: add py.typed in compliance with PEP 561
Author: niqodea
Pull request: yes

# What does this PR do?

Analogously to what has been done in [this PR](https://github.com/huggingface/diffusers/pull/5326) for diffusers, do the same for transformers.


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed.


Comments:
Hi @niqodea, thanks for opening this PR! 

Transformers doesn't officially support type checking in the library. Type annotations are used as documentation but we don't guarantee that they are consistent and checked across the repo. See: #18485 

Ah, just noticed that there has already been an attempt to integrate py.typed previously ([link](https://github.com/huggingface/transformers/pull/18485)). However, I currently don't see much complaining in my project as of now. Maybe we can try integrating it again? Feel free to close this otherwise.

@amyeroberts yep, makes sense. Thanks for your review!
fix docstring in get_oneformer_resize_output_image_size func
Author: wesleylp
Pull request: yes

# What does this PR do?
fix docstring in get_oneformer_resize_output_image_size func

## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?
@amyeroberts 


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_27207). All of your documentation changes will be reflected on that endpoint.
added unsqueeze_dim to apply_rotary_pos_emb
Author: ShashankMosaicML
Pull request: yes

# Making the unsqueeze dimension parameterized in the apply_rotary_pos_emb function in modeling_llama.py 

This PR introduces a new parameter to the [apply_rotary_pos_emb function](https://github.com/huggingface/transformers/blob/08a2edfc6629a323effd7a85feafed9e6701e2dd/src/transformers/models/llama/modeling_llama.py#L208) which allows the specification of the dimension along which to unsqueeze to make the cosine and sine rotary tensors broadcastable to the query and key tensors. This will make the function compatible with codebases that have different shapes for the query and key tensors without needing any back-and-forth transposing.

Fixes #26948


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests), Pull Request section?
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case. [Link to the issue](https://github.com/huggingface/transformers/issues/26948)
- [x] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

@gante , @ArthurZucker 



Comments:
Thank you @gante for reviewing and providing me the steps to fix the errors! However I still see some errors after following the steps, and on clicking the details for the error, I see the following. Could you please let me know how to fix this error? Thank you!
<img width="1386" alt="Screenshot 2023-10-28 at 10 16 16 PM" src="https://github.com/huggingface/transformers/assets/144760128/f8202adb-a483-4e87-b855-146af4eedad7">


Hi @amyeroberts , thanks for suggesting the changes. I have incorporated those, but  some quality checks are still failing. Could you take a look?
<img width="1055" alt="Screenshot 2023-10-30 at 2 10 24 PM" src="https://github.com/huggingface/transformers/assets/144760128/e671134a-9c70-486c-ab76-d358aa935a54">


@ShashankMosaicML running `make fixup` then committing the changes doesn't fix it?

@gante , I think that worked! (I wasn't running `make fixup` properly earlier 😅 )

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_27117). All of your documentation changes will be reflected on that endpoint.
improving TimmBackbone to support FrozenBatchNorm2d
Author: rafaelpadilla
Pull request: yes

# What does this PR do?

The `TimmBackbone` offers various customizable parameters, such as `num_channels`, `features_only`, and `out_indices`.

Currently, we lack direct support for timm's `freeze_batch_norm_2d(...) `function, which substitutes the backbone's `BatchNorm2d` layers with `FrozenBatchNorm2d`.

For models like RTDetr, Detr, ConditionalDetr, DeformableDetr, etc that require freezing of batch norm layers, one currently needs to either reimplement and modify the backbone manually or use the `timm.layers.freeze_batch_norm_2d(...)` function, like so:

```python
my_backbone = TimmBackbone.from_pretrained("resnet50d")
my_backbone = timm.layers.freeze_batch_norm_2d(my_backbone)
```
To simplify this, this PR introduces the `TimmBackboneConfig.freeze_batch_norm_2d` parameter, which defaults to False for backward compatibility. When creating the backbone, if this parameter is `True`, it will apply `timm's freeze_batch_norm_2d(...) ` function.

This update simplifies the process of creating backbone models. For models requiring a switch from `BatchNorm2d` to `FrozenBatchNorm2d`, the `TimmBackbone` can be used directly, eliminating additional steps.




## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

@amyeroberts 


Comments:
_The documentation is not available anymore as the PR was closed or merged._
Add exllamav2 better
Author: SunMarc
Pull request: yes

# What does this PR do ?

This PR is a modified version of this [PR](https://github.com/huggingface/transformers/pull/26437) that make `disable_exllama` go through a deprecation cycle. 

I also fixed the following test `test_device_and_dtype_assignment` that broke other tests in the CI introduced by this [PR](https://github.com/huggingface/transformers/pull/26761). 

I confirm that all the tests are green


Comments:
_The documentation is not available anymore as the PR was closed or merged._

Since @ArthurZucker is out next week, it would be great if you could review this PR @amyeroberts. I'm trying to have this PR in the next release. In this modified version, I make sure to deprecate `disable_exllama` arg in favor of `use_exllama`. 

Thanks for the review @amyeroberts . I've addressed all the points. LMK if something is missing ! 

Thanks for the deep review @amyeroberts ! I've added the input logic and simplified the link with optimum config. 

Thanks again @amyeroberts for iterating on this PR in such a short time !
Fix CPU offload + disk offload tests
Author: LysandreJik
Pull request: yes

Passing to safetensors serialization by default highlighted a few issues that we have with safetensors.

This PR fixes the issue, which is principally linked to weight sharing. 

Comments:
_The documentation is not available anymore as the PR was closed or merged._

@amyeroberts @patrickvonplaten if you feel uneasy with merging this right before the release, I'm fine with reverting the safetensors serialization by default to let it sit on `main` for a while longer. The release is going to be very packed already so it's fine for me.

@LysandreJik The change LGTM and seems to address some underlying issues. Re default safetensors serialization, I'm happy for it to be part of this release as long as some of the slow tests on the most popular models (bert, llama, wav2vec2, whisper, clip etc.) are good.  

Thanks both for your reviews! I'll go ahead and merge this, sorry but you'll have the conflict Patrick :grin: