noamaanMulla-03/transformers-issues huggingface dataset:
[deepspeed] check whether model is NLP one instead of counting on input type
Author: izapolsk
Pull request: yes

# What does this PR do?
This PR intends to fix an issue when training of NLP model fails if input dtype isn't int64.
My dataset had dtype = int32. Everything was ok until I decided to add deepspeed. 
It turned out that trainer relies on dtype and does input data convertion into hf_deepspeed_config.dtype if it isn't int64.
I guess it has to check whether first layer isn't Embedding instead. 

I think this PR also needs tests but I need an advice on how we can cover this case.  

@stas00 could you be so kind and review this PR and give an advice on whether tests are necessary and their implementation ?

Comments:
_The documentation is not available anymore as the PR was closed or merged._

@izapolsk, I think the fix is much simpler. Just check if it's any type of `int` - I didn't think anybody would use a different int type than int64 when I coded this, so as new use cases come in, we can just adapt for it. 

```
            if self.deepspeed and data.dtype not in [torch.in32, torch.int64]:
                # NLP models inputs are int32 or int64 and those get adjusted to the right dtype of the
```

If it resonates and works you can go ahead and apply that fix, or I can do it as well. It'd be easier for you since you have an application you can already test with.

and if this will be the way, I don't think we need any additional tests.

Actually, I wonder why were we using int64 in the first place when vocabs are so small. int32 should work always and for smaller vocabs even int16 should be enough (max `32767`). 

Probably since there is very little saving in using a more compact dtype as inputs, if they are tokenized on the fly are very short.

@stas00, done.
I added more sophisticated check based on checking first layer because vocab could be int16 - 64 and there could be other non NLP models I'm not aware of having int input. 

Thank you for reviewing this PR.

Let me re-run the offline tests first

Nope, the tests were failing. The logic was incorrect. I pushed the fix.

@izapolsk, please check that with my fix it still works for you and then we can merge it.

Thank you.

p.s. Also since it seems that this is not the last of your deepspeed improvements, here is how you can test that your future PRs work:

```
RUN_SLOW=1 pytest tests/deepspeed
```

this is because CircleCI has no GPUs, so we only run those tests requiring gpus on a different CI nightly. 

Oh good catch, missed that not. We'll need a quick rebase on main to get the quality job passing if possible (the failure seen here is fixed on main).

good catch, my bad, sorry. I'll do
Change the way tensor is reshaped in BartAttention (from .view to .reshape)
Author: raghavanone
Pull request: yes

# What does this PR do?

Fixes #21813 

Comments:
@younesbelkada For some reason fix-copies is not fixing the prophetnet copy, Not sure how to fix this.

_The documentation is not available anymore as the PR was closed or merged._

Hi @raghavanone !
Hum very weird, can you try to merge this branch with the `main` branch of `transformers` and see if this fixes the issue?

> Hi @raghavanone ! Hum very weird, can you try to merge this branch with the `main` branch of `transformers` and see if this fixes the issue?

Indeed weird, It is already on top of the latest main. Stranger thing is both of these checks pass on my machine.

Can you try `pip install --upgrade -e .["quality"]` + `make fixup`  + `make fix-copies` ?
[`Blip`] Fix blip doctest
Author: younesbelkada
Pull request: yes

# What does this PR do?

This PR fixes the Blip doctest that was failing with the changes proposed in https://github.com/huggingface/transformers/pull/21811 

Link to failing job: https://github.com/huggingface/transformers/actions/runs/4299412591/jobs/7494589393

## Why this fix is relevant? 

In #21811 the logic of `BlipForConditionalGeneration` forward pass has changed. If a user wants to use this as a standalone class and call `forward`, the text input must be fed to the model to the text decoder to mimic the implementations of encoder-decoder architectures in `transformers`, check for instance what is done to properly call `forward` on `T5`: https://github.com/huggingface/transformers/blob/b29e2dcaff114762e65eaea739ba1076fc5d1c84/src/transformers/models/t5/modeling_t5.py#L1641 

Hence, the fix of the doctest should be to feed a text input to the decoder by adding a text argument to the processor.

cc @ydshieh @sgugger 💯 

Comments:
_The documentation is not available anymore as the PR was closed or merged._
Fix page counting in Slack CI report script.
Author: ydshieh
Pull request: yes

# What does this PR do?

We started having an issue getting slack report for our CI. The error shown at the end indicates there is an issue to get all the job links for a workflow run. I take a look and find a change is necessary.

I am not sure why it was that way, especially for `i+2` part. But I remembered it has to be that to avoid duplicated pages. Maybe GitHub Actions change their API now and causes the issue.


```bash
    "url": f"{github_actions_job_links['Extract warnings in CI artifacts']}",
KeyError: 'Extract warnings in CI artifacts'
```

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_21795). All of your documentation changes will be reflected on that endpoint.

My self doubt sprit and intuition led me to look the issue again, and it turned out the issue was coming from the GitHub API call rate limit was reached as we didn't use a token when making these calls. PR #21823 was opened and merged.

The current way of page counting was good - the first page could be `0`, `1` or without page number. The next one would be `2`. My math capability was reduced a lot, especially when I tried to quickly fix things on Friday.
Add an utility file to get information from test files
Author: ydshieh
Pull request: yes

# What does this PR do?

**Add an utility file to get information from test files.**

There are more places where we need to access the information contained in test files:

- tiny model creation script: need model tester in order to create (tiny) configuration
- pipeline testing:
  - need some information so we can create `pipeline_model_mapping` in a systematic way (to avoid human error, to avoid time consuming manual edits) from the existing `model_mapping` and `tf_model_mapping` under `xxxPipelineTests` classes (which use AUTO mappings)
  - need information so we can develop some checks to make sure no pipeline tests are missing

I think it's good if we have a centralized place (file) providing ways to get this information, therefore this PR comes.
This new file will be under development.

## One example usage

### code snippet
```
test_file = "tests/models/blip/test_modeling_blip.py"
test_file = f"{os.path.sep}".join(test_file.split("/"))

model_test_mapping = get_model_to_test_mapping(test_file)
model_tester_mapping = get_model_to_tester_mapping(test_file)

print(json.dumps(to_json(model_test_mapping), indent=4))
print(json.dumps(to_json(model_tester_mapping), indent=4))
```

### model to test classes
```python
{
    "BlipForConditionalGeneration": [
        "BlipTextImageModelTest"
    ],
    "BlipForImageTextRetrieval": [
        "BlipTextRetrievalModelTest"
    ],
    "BlipForQuestionAnswering": [
        "BlipTextImageModelTest",
        "BlipVQAModelTest"
    ],
    "BlipModel": [
        "BlipModelTest"
    ],
    "BlipTextModel": [
        "BlipTextModelTest"
    ],
    "BlipVisionModel": [
        "BlipVisionModelTest"
    ]
}
```

### model to tester classes
```python
{
    "BlipForConditionalGeneration": [
        "BlipTextImageModelsModelTester"
    ],
    "BlipForImageTextRetrieval": [
        "BlipTextRetrievalModelTester"
    ],
    "BlipForQuestionAnswering": [
        "BlipModelTester",
        "BlipTextImageModelsModelTester"
    ],
    "BlipModel": [
        "BlipModelTester"
    ],
    "BlipTextModel": [
        "BlipTextModelTester"
    ],
    "BlipVisionModel": [
        "BlipVisionModelTester"
    ]
}
```

Comments:
_The documentation is not available anymore as the PR was closed or merged._

> It's all looking great! Can you add a couple of unit tests in `tests/repo_utils` for this file? It may help you catch some bugs and make sure any future PRs don't break anything.

Hi! I added a test file under `tests/repo_util/`, but as you know, the newly introduced methods require some libraries to be there, so we can import modules and get the list of model tester/test/classes. Therefore we can't test against some expected values on CircleCI `repo_utils_job`, where even `torch` or `vision` is not there.

Do you have any suggestion, say, moving this new test file outside `tests/repo_util/`?

~~Or maybe I should create some dummy model/test/tester classes dynamically inside the test file, and use them for testing?~~
Running summarization with default model fails. 4.27.0.dev0
Author: jojivk73
Pull request: no

### System Info

When runining examples/tensorflow/summarization/run_summarization.py
as given in README, 
python run_summarization.py  \
--model_name_or_path facebook/bart-base \
--dataset_name cnn_dailymail \
--dataset_config "3.0.0" \
--output_dir /tmp/tst-summarization  \
--per_device_train_batch_size 8 \
--per_device_eval_batch_size 16 \
--num_train_epochs 3 \
--do_train \
--do_eval


it fails as below.

![image](https://user-images.githubusercontent.com/14943401/222041742-82660acf-6fa4-40a4-af92-38a81044ddf0.png)



### Who can help?

_No response_

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

1. Copy script in README to a file
2. Run the script

### Expected behavior

Should run without issues as it is example given in README

Comments:
cc @Rocketknight1 

Hi @jojivk73, thanks for the bug report! We've reproduced the issue - the cause is that the `transformers` library is currently transitioning to using standardized native Keras layers for as many purposes as possible, and deprecating the previous setup where we often had ad-hoc model-specific solutions. 

One consequence of the transition is that BART's embeddings used to store their weights in `embeddings.weight`, but now that they've been swapped to a Keras `Embedding` layer, the weights are in `embeddings.embeddings`. We missed this issue in the example code during the transition, but we're preparing a PR to fix it immediately. I'll ping you as soon as it's ready.

@jojivk73 the PR is now up at #21881

@jojivk73 PR is merged. Please install the latest version from `main` and let me know if you have any other problems, and thanks again for the bug report!
Add check for different embedding types in examples
Author: Rocketknight1
Pull request: yes

This PR fixes an issue that occurred because we did two things at the same time: We started transitioning our models to use `keras.Embedding` layers, but also added code to the examples to only resize embeddings when necessary by checking `model.get_input_embeddings().weight.shape`. However, because of the transition, the relevant variable became `model.get_input_embeddings().embeddings` in some cases!

To fix this, I added a check for both types of embeddings. When the transition is complete, we can remove the `.weight` code path and only use `.embeddings`.

I've checked that all the affected examples run without errors using the command supplied in the README following this PR.

Fixes #21865 

Comments:
_The documentation is not available anymore as the PR was closed or merged._
Add TFVisionTextDualEncoder
Author: Rocketknight1
Pull request: yes

This PR uses the new weight crossloading functions to add the missing `TFVisionTextDualEncoder` class.

Comments:
_The documentation is not available anymore as the PR was closed or merged._

Good spot, I've added your suggestions and I'll add the modeling file to the documentation check list!

The failing test is unrelated (OPT generation), merging!
Fix `WhisperModelTest` 
Author: ydshieh
Pull request: yes

# What does this PR do?


### Fix `test_model_parallelism` for `Whisper`
A merged PR `Add WhisperTokenizerFast` (#21222)  (02/21) started failing `test_model_parallelism` .
That PR didn't change relevant test/modeling file, but just changed the model tester's vocab size from `99` to `200`.

When I traced it, I found at this places inside `WhisperDecoder`
```python
inputs_embeds = self.embed_tokens(input_ids)
positions = self.embed_positions(input_ids, ...)
hidden_states = inputs_embeds + positions
```
- in previous commit,  `embed_tokens` and `embed_positions` weight matrices are on the same GPU `1`.
- After that PR, one is on GPU `0` another being on GPU `1`. 

I fixed the issue by adding

```python
    # Needs higher percentages after model tester's vocab_size is changed to 200 (PR #21222)
    model_split_percents = [0.8, 0.9]
```

### Fix `test_torchscript_*` for `Whisper`

PR #21298 added an optional argument `attention_mask` to `WhisperModel` model classes. `torchscript` tests need a bit change to make it work.

Comments:
_The documentation is not available anymore as the PR was closed or merged._
Text-classification example does not work as is on 4.27.0.dev
Author: jojivk73
Pull request: no

### System Info

Transformer : 4.27.0.dev
Example in : examples/tensorflow/text-classification/run_glue.py 
fails when running example given in README.

"""
python run_glue.py \
--model_name_or_path distilbert-base-cased \
--task_name mnli \
--do_train \
--do_eval \
--do_predict \
--output_dir  outdir \ 
--predict_file data_to_predict.json
"""

Issues : 
  1.  missing data_to_predict.json>
  2. output_dir not mentioned in README.
  2. Fails with following error without --predict_file + --output_dir
  
 
![image](https://user-images.githubusercontent.com/14943401/222235813-1fb96343-9e3d-461a-bd83-039ecbf814e5.png)

  
  

### Who can help?

_No response_

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

Run the script given in description 

### Expected behavior

The task run without issues.

Comments:
cc @Rocketknight1 

My bad :(.  It was trying to use for GPT-J  instead of distilbert-base-cased

Ah yes - CLM models generally need some modifications to transfer to text classification tasks, including adding a padding token. Their performance is also usually worse. Using a MLM (masked language model, like BERT/RoBERTa/DistilBERT/DeBERTa) base instead will work much better!
Fix pipe comm test
Author: ydshieh
Pull request: yes

# What does this PR do?

PR #21600 added task `zero-shot-audio-classification`:
```python
    "zero-shot-audio-classification": {
        "impl": ZeroShotAudioClassificationPipeline,
        "tf": (TFAutoModel,) if is_tf_available() else (),
        "pt": (AutoModel,) if is_torch_available() else (),
        "default": {
            "model": {
                "pt": ("laion/clap-htsat-fused", "f39917b"),
            }
        },
        "type": "multimodal",
    },
```

But this fails the test `tests/pipelines/test_pipelines_common.py::PipelineUtilsTest::test_load_default_pipelines_tf` as there is `tf` key under the `task`, but not under `task/default`. There is a check in that test method:
```python
        if len(relevant_auto_classes) == 0:
            # task has no default
            logger.debug(f"{task} in {framework} has no default")
            return
```
So I decide to set `"tf": (),` for now. 

(We have no `TFClap` yet)

Comments:
_The documentation is not available anymore as the PR was closed or merged._
[ZAC] fix ci daily 
Author: ArthurZucker
Pull request: yes

# What does this PR do?
Fixes the loading of zero shot audio classification pipeline by providing a correct revision. The previous one was destroyed when checkpoints were overwritten at some point. (that's also when the model card dissapeared). 

Comments:
_The documentation is not available anymore as the PR was closed or merged._

Failing test is unrelated
`shuffle` argument when initializing Sampler-related classes in `trainer.py`
Author: gugarosa
Pull request: no

### Feature request

Add support for an additional keyword argument (`shuffle`) in `DistributedSampler` (https://github.com/huggingface/transformers/blob/main/src/transformers/trainer.py#L839).

### Motivation

It samples data in a deterministic way across multiple processes. However, when using a contiguous array of pre-encoded data, using a shuffler in the sampler could avoid sequential input_ids for extreme long files and possibly diminish the number of spikes in the loss.

### Your contribution

Would you consider worth in adding it? In my use case, I have to re-write the whole `_get_train_sampler()` just to set `shuffle=True`.

If necessary, I can contribute with a PR.

Thanks for your attention and best regards,
Gustavo.

Comments:
`shuffle=True` is the default for this DistributedSampler (see [doc](https://pytorch.org/docs/stable/data.html#torch.utils.data.distributed.DistributedSampler)). There is no need to pass it.

You are right, my bad. I overlooked the documentation.
`import decord` crashes Python kernel/process when moving X-CLIP or other video classification models to CUDA GPU
Author: e-caste
Pull request: no

### System Info

- `transformers` version: 4.25.1
- Platform: Linux-6.0.12-76060006-generic-x86_64-with-glibc2.35
- Python version: 3.10.6
- Huggingface_hub version: 0.11.1
- PyTorch version (GPU?): 1.13.0+cu117 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: yes
- Using distributed or parallel set-up in script?: no

### Who can help?

@sgugger for the docs and @NielsRogge for X-CLIP specifically.

### Information

- [X] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

I was trying to run inference on GPU with X-CLIP on my own dataset. To do so I followed the [example code](https://huggingface.co/docs/transformers/main/en/model_doc/xclip#transformers.XCLIPModel.forward.example) in the docs, which uses decord as library to load videos into memory. I tested it and it ran perfectly, _but_ I noticed the model was on CPU.  
A simple `model.to(torch.device("cuda"))` ought to do the trick, right? Well no, here is the rabbit hole I went down: https://github.com/huggingface/transformers/issues/21054.

My conclusion is that a simple `import decord` before trying to load the model into GPU memory is enough to make the `python` process crash, be it in the terminal or a Jupyter kernel.  
This is both with the latest decord version from PyPI (which can only run on CPU) and with the latest decord version compiled from source with CUDA enabled.

To fix this, I used the [code](https://colab.research.google.com/gist/nateraw/c327cb6ff6b074e6ddc8068d19c0367d/pyav-io.ipynb#scrollTo=fzGRpWaUqnTL) generously provided by @nateraw built on pyAV, while discussing how to integrate videos in the datasets library (https://github.com/huggingface/datasets/issues/5225).

### Expected behavior

The model should be transferable to the GPU without issue.  
To instruct people on how to do so, the docs should be updated to make use of pyAV instead of decord to avoid sending other users into hours if not days of debugging.

Comments:
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

Unstale

cc @amyeroberts
Use PyAV instead of Decord in examples
Author: amyeroberts
Pull request: yes

# What does this PR do?

Replaces decord with pyav as the library to decode video files in examples. This resolves an issue where the CUDA crashes if `decode` has been imported. 

The doc tests have been run to validate they pass cc @ydshieh 

Note: decord is still being used in the video pipelines. cc @nateraw 

Fixes #21085

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [x] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

Comments:
_The documentation is not available anymore as the PR was closed or merged._

Thanks for the review @nateraw! I've updated the docstrings with your suggestions :)
Add `inputs_embeds` functionality when generating with BioGPT 
Author: sidkiblawi
Pull request: yes

# What does this PR do?


This PR extends https://github.com/huggingface/transformers/pull/21405 by @gante to BioGPT, making it accept inputs_embeds when generating.

```
import torch
from transformers import  BioGptTokenizer, BioGptForCausalLM


model = BioGptForCausalLM.from_pretrained("microsoft/biogpt")
tokenizer = BioGptTokenizer.from_pretrained("microsoft/biogpt")

inputs_embeds = torch.rand((1, 1, 1024)) # 1 dummy soft-prompt token embeddings
attention_mask = torch.ones(inputs_embeds.shape[:2], dtype=torch.long)
filler_input_ids = torch.LongTensor([[model.config.bos_token_id]])

model.generate(filler_input_ids,attention_mask = attention_mask, inputs_embeds=inputs_embeds, max_new_tokens=300, num_beams=4)
```
# Who can Review?
@gante 


Comments:
_The documentation is not available anymore as the PR was closed or merged._

review request: @gante 

Thanks for your contribution!