noamaanMulla-03/transformers-issues huggingface dataset:
Fix id2label assignment in run_classification.py
Author: jheitmann
Pull request: yes

# What does this PR do?

This pull request addresses an issue in the `run_classification.py` script where the assignment of the `id2label` attribute in the model's config is incorrect. The current implementation copies `config.label2id` without modifying it, leading to an incorrect mapping. The proposed fix ensures that the `id2label` attribute is assigned based on the correct mapping (`label_to_id`) to resolve this issue.

**Changes Made:**
- Modified the assignment of `id2label` in the `run_classification.py` script to use the correct label-to-id mapping.

**Context:**
This issue was introduced with transformers version 4.36, and the incorrect assignment can lead to unexpected behavior in the script.

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

Fixes #28589 


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [X] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

- @ArthurZucker  
- @younesbelkada 


Comments:
Fix id2label assignment in run_classification.py
Author: jheitmann
Pull request: no

### System Info

- `transformers` version: 4.36.2
- Platform: macOS-13.4-arm64-arm-64bit
- Python version: 3.10.10
- Huggingface_hub version: 0.20.2
- Safetensors version: 0.4.0
- Accelerate version: 0.23.0
- Accelerate config:    not found
- PyTorch version (GPU?): 2.1.0 (False)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: no
- Using distributed or parallel set-up in script?: no

### Who can help?

@ArthurZucker @younesbelkada 

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

```
export TASK_NAME=mrpc

python examples/pytorch/text-classification/run_glue.py \
  --model_name_or_path bert-base-cased \
  --task_name $TASK_NAME \
  --do_train \
  --do_eval \
  --max_seq_length 128 \
  --per_device_train_batch_size 32 \
  --learning_rate 2e-5 \
  --num_train_epochs 3 \
  --output_dir /tmp/$TASK_NAME/
  ```

### Expected behavior

**Issue Description:**
The `run_classification.py` script currently has an issue where the assignment of `id2label` in the model's config is incorrect. The problem arises from copying `config.label2id` without modifying it later on. This issue was introduced with transformers version 4.36.

**Steps to Reproduce:**
1. Execute the `run_classification.py` script with a configuration file.
2. Inspect the `id2label` attribute in the model's config.

**Expected Behavior:**
The `id2label` attribute should be assigned correctly, reflecting the label-to-id mapping.

**Actual Behavior:**
The `id2label` attribute is assigned based on the original `config.label2id`, leading to incorrect mapping.

**Proposed Solution:**
Modify the following line in `run_classification.py`:

```python
model.config.id2label = {id: label for label, id in config.label2id.items()}
```

to:

```python
model.config.id2label = {id: label for label, id in label_to_id.items()}
```

This change ensures that the correct mapping is used.

Comments:
model.config.id2label = {id: label for label, id in config.label2id.items()}
Is there some bug in typehint in `modeling_outputs` (or maybe other files)?
Author: haotong-yang
Pull request: no
Labels: name: Good First Issue

### System Info

Hi! When I tried to modify some code which calls some classes in `src/transformers/modeling_outputs.py` such as the class `BaseModelOutput` and `BaseModelOutputwithPast`, I find the typehint of some parameters is different from your comments and my understanding of the code. I believe the correct type of them should be like `Optional[Tuple[torch.FloatTensor, ...]]` instead of `Optional[Tuple[torch.FloatTensor]]` because the comments say the tuple with length which equals to the num of layers or something else but not **ONE**. However, the type hint `Optional[Tuple[torch.FloatTensor]]` means the tuple should be length 1.

Is my understanding correct? Or am I misunderstanding what these codes do?

![image](https://github.com/huggingface/transformers/assets/56245046/90295155-0ad3-4f30-b771-aeeee2179740)

@ArthurZucker @younesbelkada Thank you!

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

I use the Pylance as the type checker.

### Expected behavior

The typehint would not affect the behavior.

Comments:
Hey! I think that we used Tuple[X] when there can be N instances of X. I don't mind using ellipsis. 
cc @Rocketknight1 who is our type expert!  

@ArthurZucker Thank you for your reply! I am a new guy for type hint so I am not sure about the situation about the previous version (before Python 3.10) but in the lastest version (Python 3.11 or 3.12), the Tuple[X] just means a tuple with **ONE** element with type X. 
![image](https://github.com/huggingface/transformers/assets/56245046/0fcfec13-bb78-4e27-adfe-d992ee831508)
This is an example provided by the document of Python: https://docs.python.org/3/library/typing.html
And my Pylance checker has indeed raise a warning about it.

And also the type hint `Tuple` and `tuple` are considered as the same:
![image](https://github.com/huggingface/transformers/assets/56245046/31f1656c-8661-41c4-88ae-016e7a853f97)


feel free to open a PR for a fix if you want 🤗 

Hi @gary-young, I think you're correct! We realize it would be quite a significant job to update all instances of this across the codebase, but if you want to contribute fixes to one or more files, we'd be happy to accept a PR.

Ok! I am willing to fix some of them for those I can identify.
Add config tip to custom model docs
Author: Rocketknight1
Pull request: yes

Mentioned this to @LysandreJik earlier - this PR adds a tip to the docs on uploading custom code models to encourage users to use a monolithic config that gets passed to sub-layers, like we use in core `transformers` code. Some models that didn't do this were very painful to port and required rewrites to all layers, so encouraging users to do this earlier might help a lot.

Comments:
How to find or create the `model_state_dict.bin` file for the `convert_llava_weights_to_hf.py` script
Author: isaac-vidas
Pull request: no

Hi @younesbelkada,

Following up on the [fix to the LLaVA convert script](https://github.com/huggingface/transformers/pull/28570) and thanks for all the help with the PR!

I encountered some issue with the convert script and wanted to ask about the recommended way to create the `model_state_dict.bin` file specified here: https://github.com/huggingface/transformers/blob/772307be7649e1333a933cfaa229dc0dec2fd331/src/transformers/models/llava/convert_llava_weights_to_hf.py#L74

In order to create the `model_state_dict.bin` I tried something like the following with the original https://github.com/haotian-liu/LLaVA code:
```python
import torch
from llava.model.language_model.llava_llama import LlavaLlamaForCausalLM

# load model
kwargs = {"device_map": "auto", "torch_dtype": torch.float16}
model = LlavaLlamaForCausalLM.from_pretrained("liuhaotian/llava-v1.5-7b", low_cpu_mem_usage=True, **kwargs)

# load vision tower
model.get_vision_tower().load_model()

# Save state dict
torch.save(model.state_dict(), "tmp/hf_models/llava-v1.5-7b/model_state_dict.bin")
```

It works but when I used the convert script I had to make the following changes: 
* Remove keys that ended with `.inv_freq` (e.g. `language_model.model.layers.0.self_attn.rotary_emb.inv_freq`)
* Comment out the update to the `model.config.vocab_size` and `model.config.text_config.vocab_size` with the `pad_shape` here: https://github.com/huggingface/transformers/blob/772307be7649e1333a933cfaa229dc0dec2fd331/src/transformers/models/llava/convert_llava_weights_to_hf.py#L96-L97 otherwise, when I would try to load the converted model, it will error with the following:
    ```python
    from transformers import AutoProcessor, LlavaForConditionalGeneration
    model_id = "Shopify/llava-1.5-7b"

    model = LlavaForConditionalGeneration.from_pretrained(
        model_id,
        torch_dtype=torch.float16,
        low_cpu_mem_usage=True,
    ).to(0)
    ```
    ```console
    ValueError: Trying to set a tensor of shape torch.Size([32064, 5120]) in "weight" (which has shape torch.Size([32128, 5120])), this look incorrect.
    ```

Am I doing something wrong when I create the `model_state_dict.bin` file or am I missing something else?

Thanks again in advance.

Comments:
Hi @isaac-vidas 
Thanks very much for your message, I think that these [two lines](https://github.com/huggingface/transformers/blob/772307be7649e1333a933cfaa229dc0dec2fd331/src/transformers/models/llava/convert_llava_weights_to_hf.py#L96-L97) seems wrong, does replacing them with `model.get_input_embeddings().weight.shape[0]` fixes the issue? 

You could also just remove these two lines as `resize_token_embeddings` method applies the changes to the config already

P.S. Also had to remove `self_attn.rotary_emb.inv_freq` keys to perform the conversion from the checkpoint uploaded to the hub

Added a PR related to the discussion above https://github.com/huggingface/transformers/pull/28617
Bug in GPT NeoX Implementation
Author: andersonbcdefg
Pull request: no

### System Info

GPT-Neo-X does not have a "q_proj" module, so the following lines that check for the dtype raise an error.

```
input_dtype = query.dtype
        if input_dtype == torch.float32:
            if torch.is_autocast_enabled():
                target_dtype = torch.get_autocast_gpu_dtype()
            # Handle the case where the model is quantized
            elif hasattr(self.config, "_pre_quantization_dtype"):
                target_dtype = self.config._pre_quantization_dtype
            else:
                target_dtype = self.q_proj.weight.dtype
```

This is in `transformers/src/transformers/models/gpt_neox/modeling_gpt_neox.py`

### Who can help?

text models: @ArthurZucker and @younesbelkada

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

1. Initialize a GPT Neo X model with flash attention
model = AutoModel.from_pretrained(model_name_or_path, attn_implementation="flash_attention_2", torch_dtype=torch.bfloat16)

2. Try to run forward pass. Causes following error: 
AttributeError: 'GPTNeoXFlashAttention2' object has no attribute 'q_proj'

### Expected behavior

The forward pass should work, or at least fail for a reason other than a reference a module that the given model does not actually have

Comments:
Nice catch ! #28645 should fix it
Remove CaptureLogger
Author: amyeroberts
Pull request: yes

# What does this PR do?

Replace our custom CaptureLogger class with the python library `unittest.TestCase().assertLogs`. 

Makes testing for no logs being raised more robust - asserts no logs rather than string matching. 


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_28540). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Closing for now as we need some funky logs for the no logs case whilst we still support < 3.10 and the logger issue has been resolved.
Support gated Linear Layers for SwitchTransformers
Author: agemagician
Pull request: yes

# What does this PR do?

The new version of SwitchTransformers uses a gated linear layer.

This pull request adds gated Linear Layers support for SwitchTransformers.

This is very similar to T5 and T5 v1.1 models.


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [X] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

@amyeroberts , @ArthurZucker , @younesbelkada , @younesbelkada , @sgugger

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_28587). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

> Looks good I would say ! Do you know if there are any checkpoint on the Hub that uses gated linear layers?

Yes, I am currently training a new model and have the checkpoint privately.
I will release it soon after finishing training and testing :)

> Thanks, not really sure what the incentive is for that? This is against our philosophy + no checkpoints use this no?

Hi @ArthurZucker ,

I have a new large model that was trained on the latest version of the switch-transformer, and it uses the gated method.
I will release the model after training is finished and it is fully tested on downstream tasks. 

The current implementation is exactly the same as T5 and T5v1.1 on huggingface :

https://github.com/huggingface/transformers/blob/v4.36.1/src/transformers/models/t5/modeling_t5.py#L333


@agemagician Thanks for your contribution! What I'd suggest is having this model code on the hub. This way you or anyone else in the community can use it. If we see many checkpoints  using this, or users requesting it, then we can consider integrating it into the transformers repo. As @ArthurZucker points out, adding custom changes to our modeling code without checkpoints and increased if/else logic doesn't align with the repo's philosophy. 

OK, no problem. I will close this pull request and add it to the hub.
Use mmap to accelerate checkpoint loading
Author: weimingzha0
Pull request: no

### Feature request

Use torch.load(mmap=True) if possible.





### Motivation

Python 2.1 allows mmap() when loading checkpoints ([doc](https://pytorch.org/docs/stable/generated/torch.load.html))
I tested on a 6B model: with mmap(),   takes 2.x seconds to load (vs 12.x seconds without using mmap)

### Your contribution

https://github.com/huggingface/transformers/pull/28331

Comments:
Closing as the PR was merged
Fix phi model doc checkpoint
Author: amyeroberts
Pull request: yes

# What does this PR do?

Small fix c.f. https://github.com/huggingface/transformers/commit/d93ef7d7512e79612606f29e6ae308920f0a86cd#r137345713


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_28581). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
KOSMOS-2, finding probability distribution of the text sequence
Author: snpushpi
Pull request: no

### System Info

Google Colab with gpu environment enabled and related libraries installed  

### Who can help?

@ydshieh 

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below

### Reproduction

Hi, 
I am trying to extract the probability distribution of the tokens in a sequence from the model. Here is what 
`
from PIL import Image
import requests
from transformers import AutoProcessor, Kosmos2ForConditionalGeneration
import torch

model = Kosmos2ForConditionalGeneration.from_pretrained("microsoft/kosmos-2-patch14-224")
processor = AutoProcessor.from_pretrained("microsoft/kosmos-2-patch14-224")

prompt1 = "An old man sitting on a bench in a public park alone, reading a book."
url1 = "http://images.cocodataset.org/val2017/000000264535.jpg"
image1 = Image.open(requests.get(url1, stream=True).raw)

inputs1 = processor(text=prompt1, images=image1, return_tensors="pt").to(device)

model_output = model(
    pixel_values=inputs1["pixel_values"],
    input_ids=inputs1["input_ids"],
    image_embeds=None,
    image_embeds_position_mask=inputs1["image_embeds_position_mask"],
    use_cache=True,
)
input_ids = processor.tokenizer(prompt, return_tensors = 'pt').input_ids.to(device)
input_ids.shape
# torch.Size([1, 14])

model_output.logits.shape 
# torch.Size([1, 79, 65037])
token_logits = torch.cat([model_output.logits[:,:2,:],model_output.logits[:,67:,:]],dim=1) #Is this correct?
`
So I am trying to extract the logits of the text tokens from the model output. The goal is to eventually calculate the probability of a certain word given the previous words and the image. But I am not sure of the last line where I extracted the text token logits from the model output. I did what I did since that's what the inputs1["image_embeds_position_mask"] mask looked like, but I am not sure if that is the right thing to do. Can someone confirm that?


### Expected behavior

explained above. 

Comments:
@ydshieh can you look at it? Thank you so much!

Hi, 

I think you are only interested in looking the logits for the parts of `prompt1 = "An old man sitting on a bench in a public park alone, reading a book."`.

This part is after the token (id) of `end of image` (eoi, `64004`). So you have to find where the token id `64004` is in `inputs1["input_ids"]`, and only take the parts after it (along the sequence dimension).

- using `image_embeds_position_mask` works, but you don't need to having the leading 2 tokens
  - using `image_embeds_position_mask` includes `64004`, and the logits from that place is the logit for the next token (`An`) given the token of `eoi (64004)` and the previous image info etc.


Thank you, that makes sense!
OSError: /home/datascience/huggingface does not appear to have a file named preprocessor_config.json. Checkout 'https://huggingface.co//home/datascience/huggingface/None' for available files.
Author: z7ye
Pull request: no

### System Info

import transformers
transformers.__version__

'4.31.0'

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction
![image](https://github.com/huggingface/transformers/assets/25996703/6e26deb4-237b-473a-8874-164a4d134ea8)

```python
segmenter = pipeline(task="image-segmentation", model="facebook/detr-resnet-50-panoptic", revision="fc15262")
segmenter.save_pretrained("./huggingface")


from transformers import pipeline
task = 'image-segmentation'
model_dir="./huggingface"
model = pipeline(task, model = model_dir)

OSError: /home/datascience/huggingface does not appear to have a file named preprocessor_config.json. Checkout 'https://huggingface.co//home/datascience/huggingface/None' for available files.
```

### Expected behavior

no bug

Comments:
Hey! Thanks for reporting! Yep I thing we should make sure the `image_processor`is also saved! Would you like to open a PR? 🤗 

Same here, but why it need this one for LLava1.5 model?
Mixtral: Reduce and Increase Expert Models
Author: ryan-minato
Pull request: yes

This pr adds a method to MixtralSparseMoeBlock, MixtralDecoderLayer and MixtralModel that removes one or more experts from MixtralModel.

Comments:
cc @ArthurZucker @younesbelkada 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.