noamaanMulla-03/transformers-issues huggingface dataset:
Convert numpy arrays to lists before saving the evaluation metrics as json
Author: harisankar95
Pull request: yes

eval_metrics contains:
    - mean_iou: float
    - mean_accuracy: float
    - overall_accuracy: float
    - per_category_iou: ndarray of shape (num_labels,)
    - per_category_accuracy: ndarray of shape (num_labels,)
ndarrays are to be converted to lists for json serialization.


Comments:
@sgugger Can you please review this small update.

_The documentation is not available anymore as the PR was closed or merged._

> LGTM! Can you just run `make style` on your branch to fix the styling issue?

yes, the styling issue is now fixed.
Hot Fix 
Author: ydshieh
Pull request: yes

# What does this PR do?

Fix the failing GitHub Action `Update Transformers metadata` due to the missing `pytest` after PR #22987. But it's kind strange that a simple `from transformers.utils import direct_transformers_import` will need `pytest`. Maybe we need to re-think if to have `from .doctest_utils import HfDocTestParser` inside the file `transformers/utils/__init__.py`.

Comments:
_The documentation is not available anymore as the PR was closed or merged._

No need to  have this PR anymore after #23245 and the decision made there.
Each list in `nested_token_ids` can't be a complete subset of another list, but is
Author: lovodkin93
Pull request: no

### Feature request

Enable passing disjunctive constraints (https://github.com/huggingface/transformers/blob/main/src/transformers/generation/beam_constraints.py#L261)
 where one is a subset of the other

### Motivation

in the constrainted beam decoding feature, specifically in the case of disjunctive constraints, currently there is no option for one disjunctive constraint to be a subset of the other, as can be seen here:
https://github.com/huggingface/transformers/blob/main/src/transformers/generation/beam_constraints.py#L220
But, in many cases, this is exactly the case.
For example, if I wanted to consider all the inflections of the verb "sentence": ["sentence", "sentences", "sentenced", "sentencing"], then the tokenizer separates "sentenced" into ["sentence", "d"], which means that "sentence" is a subset of "sentenced".

### Your contribution

N/A

Comments:
cc @gante

Hey @lovodkin93 👋  the constraints feature in beam search is experimental, so our efforts are currently limited to fixing bugs. 

If you'd like to add the feature yourself, go for it :) 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Update docs for assigning path to all_video_file_paths
Author: hom-bahrani
Pull request: yes

# What does this PR do?

Updating a few lines in the video classification tasks guide as the way its written it seems like we are not actually iterating over the file paths, but rather the string. 

Fixes # (issue)


## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [x] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.



Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_22522). All of your documentation changes will be reflected on that endpoint.

@amyeroberts yes you're right, the dataset is downloaded to the .cache directory and extracted into local directory. I have updated accordingly and tested in colab to confirm that its working 

<img width="1387" alt="Screenshot 2023-04-05 at 16 06 57" src="https://user-images.githubusercontent.com/8465628/230124560-1eef5951-70f3-4b10-b4a8-4d1766bcb531.png">


This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Fix doctest files fetch issue
Author: ydshieh
Pull request: yes

Reverts huggingface/transformers#23271

Embarrassingly and unfortunately, the new job `tests_pr_documientation_tests` fails at the step `Get files to test` when the job is run on the `main` branch.

https://app.circleci.com/pipelines/github/huggingface/transformers/64235/workflows/54a99003-258e-4c2a-8366-b4461b3ec33f/jobs/794628/parallel-runs/0/steps/0-113

I will have to take a look - the log is not informative.

Comments:
_The documentation is not available anymore as the PR was closed or merged._

As said offline I don't think we need to revert urgently, we can just ignore the red check on main.

@sgugger FYI The doctest PR also has some problem running on our GitHub Actions runner. See error below.
I will take a look, but this PR could be merged (without fixing the following issue) once you think the changes are good.


```bash
_____ ERROR collecting src/transformers/generation/configuration_utils.py ______
import file mismatch:
imported module 'transformers.generation.configuration_utils' has this __file__ attribute:
  /transformers/src/transformers/generation/configuration_utils.py
which is not the same as the test file we want to collect:
  /__w/transformers/transformers/src/transformers/generation/configuration_utils.py
HINT: remove __pycache__ / .pyc files and/or use a unique basename for your test file modules
```

> @sgugger FYI The doctest PR also has some problem running on our GitHub Actions runner. See error below. I will take a look, but this PR could be merged (without fixing the following issue) once you think the changes are good.
> 
> ```shell
> _____ ERROR collecting src/transformers/generation/configuration_utils.py ______
> import file mismatch:
> imported module 'transformers.generation.configuration_utils' has this __file__ attribute:
>   /transformers/src/transformers/generation/configuration_utils.py
> which is not the same as the test file we want to collect:
>   /__w/transformers/transformers/src/transformers/generation/configuration_utils.py
> HINT: remove __pycache__ / .pyc files and/or use a unique basename for your test file modules
> ```

@sgugger A short fix for this issue is given in [the last commit](https://github.com/huggingface/transformers/pull/23277/commits/bada5b3d616bff32f7440408428a4b9ed13c503b).

The reason is the file `conftest.py` has this line `from transformers.testing_utils import HfDoctestModule, HfDocTestParser` added in #23271. However, the `transformers` is installed during docker image build, which is different from the one when the CI is run.

This change should be applied to other workflow file, but it's rare that we have such imports in the codebase. I will do it in a separate PR.

Thanks for all the explanation!

Going to merge as the failing tests are irrelevant and I have tried to re-run for a few times.
Seq2Seq Trainer Handling for MLFlow Exception
Author: njbrake
Pull request: no

### System Info

Transformers 4.28.1, torch 1.13.1

When using the Seq2SeqTrainer with the MLFlow integration enabled, if I lose my connection to mlflow after the training has begun (if the server crashes or if there is a network error), MLFlow throws the exception:

```
raise MlflowException(f"API request to {url} failed with exception {e}")
```

I don't know if this is a bug, or if you have a recommended way to continue: I would like the option to have the seq2seqtrainer log the mlflow connection error but then continue training with the mlflow integration disabled. 

I imagine that this behavior would apply to any integration, not just mlflow.


### Who can help?

@sgugger 

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

1. Start training with Seq2Seq Trainer with a connection to MLflow
2. During training, stop the mlflow server
3. Seq2Seq Trainer raises an MLFlow Exception

### Expected behavior

I would expect that there would be an option to allow for mlflow exception to disable the integration but continue training.

Comments:
We don't maintain the MlFlow integration ourselves, so I can't really help. Maybe try to tag the persons who added it?

Thanks for the quick reply! Although my question involves MLFlow, I think the question more broadly is:

if an integration callback throws an error, how can we disable that integration and continue with training?

I think that might be a question for the huggingface team and not for the MLFlow integrators?

Oh I dind't realize you were asking for that. This is what the [`report_to`](https://huggingface.co/docs/transformers/main_classes/trainer#transformers.TrainingArguments.report_to) argument is for :-)

Ah. Perfect. Thanks so much!
Better check for packages availability
Author: apbard
Pull request: yes

Following up huggingface/accelerate#1356
Refactor checks to avoid boilerplate and to ensure we are not picking up a folder that happens to be called as the package.

I assume all _*_available are for caching purposes. But is that really needed?

Comments:
_The documentation is not available anymore as the PR was closed or merged._

> Thanks a lot for the refactor, this is super nice!
> 
> There are a couple of packages that do not properly implement metadata (I kno for sure `opencv` does not since we are adding it in another PR), could you quickly check that all the packages for which you did this PR do implement the metadata? if they don't we need to rely on the old way, which is fine as it should be an exceptional case.

I have checked the packages and found issues only on sklearn (the import is sklearn and package scikit-learn) and decord.

Could you please double check: smdistributed, tensorflow_text, torchdistx?

I have validated that `tensorflow_text` works. Installing `torchdistx` seems to painful for the time I have right now, but looking at the GitHub, everything should be fine.

Checking internally for `smdistributed` since it only exists in SageMaker envirnoments.

Not hearing anything back on `smdistributed`, so let's just merge this PR and see if anyone complains :sweat_smile: 

Can you just fix the conflict?

don't think failures are due to this PR. Also main is failing
Add gradient_checkpointing parameter to FlaxWhisperEncoder
Author: raghavanone
Pull request: yes

Ref https://github.com/huggingface/transformers/pull/23173#discussion_r1188815621

@sanchit-gandhi 

Comments:
_The documentation is not available anymore as the PR was closed or merged._

Merging as the failing tests are also failing on main and not related to this PR
ImportError: Datasets needs to be installed if not passing speaker embeddings.
Author: pannous
Pull request: no

### System Info

- `transformers` version: 4.29.0
- Platform: macOS-13.3.1-arm64-arm-64bit
- Python version: 3.10.10
- Huggingface_hub version: 0.14.1
- Safetensors version: 0.3.1
- PyTorch version (GPU?): 2.0.0 (False)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: NO
- Using distributed or parallel set-up in script?: NO

### Who can help?

@sanchit-gandhi ?

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

Maybe not a bug but lacking documentation:

Following the simple exampe https://huggingface.co/docs/transformers/transformers_agents

```
from transformers import OpenAiAgent
from transformers import HfAgent
...
text="A beaver is swimming in the water"
audio=agent.run("Read the following text out loud", text=text)
```

```
==Explanation from the agent==
I will use the following  tool: `text_reader` to read the text out loud.


==Code generated by the agent==
audio = text_reader(text)


==Result==


│ /Users/me/.pyenv/versions/3.10.10/lib/python3.10/site-packages/transformers/tools/text_to_speech.py:52 in encode                                                                                 
│   50 │   │   if speaker_embeddings is None:                                                      │
│   51 │   │   │   if not is_datasets_available():                                                 │
│ ❱ 52 │   │   │   │   raise ImportError("Datasets needs to be installed if not passing speaker    │
```



@sanchit-gandhi ?

### Expected behavior

The example should work out of the box, or add information on how to download the required dataset in the documentation and error message.

Comments:
Thanks @pannous! Indeed, we should show what are the requirements. The error you're getting is due to `datasets` not being installed.

This PR should fix it: https://github.com/huggingface/transformers/pull/23301
Revert "search buffers for dtype"
Author: sgugger
Pull request: yes

Reverts huggingface/transformers#23159

This breaks the FDSP integration for some reason, so reverting for now as we investigate things further. The revert will be included in the patch 4.29.1

(Test that breaks:
```
RUN_SLOW=yes pytest -s -v tests/fsdp -k test_checkpointing
```
in Accelerate)

Comments:
_The documentation is not available anymore as the PR was closed or merged._
Revert DeepSpeed stuff
Author: muellerzr
Pull request: yes

# What does this PR do?

During the integration some deepspeed items that need further looks into before it can be part of the integration need to be addressed/looked at more carefully. This PR reverts the base deepspeed logic done in `setup_devices` and `parallel_mode` to restore the original deepspeed behavior

Fixes # (issue)


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

@sgugger, cc @pacman100 so you're aware. 

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_22899). All of your documentation changes will be reflected on that endpoint.

I see this was included in transformers 4.29.0 (https://github.com/huggingface/transformers/releases/tag/v4.29.0). Could you share more about how this changes the Transformers + DeepSpeed integration? I don't quite understand the diff. Does this disable some deeper level of integration of DS with Transformers?

@jli this pr just reverted a small portion of Accelerate handling the deepspeed part when we weren't ready for that yet. CC @pacman100 if you could explain accelerates deepspeed integration vs the transformers one were replacing in terms of features? :)
Add  document-question-answering in task_summary
Author: y3sar
Pull request: yes

# What does this PR do?

From issue #18926
This PR adds Document Question Answering summary to task_summary.mdx
It also provides an example using pipeline and this [model](https://huggingface.co/naver-clova-ix/donut-base-finetuned-docvqa)  

## Who can review?
@stevhliu 





Comments:
_The documentation is not available anymore as the PR was closed or merged._

@stevhliu I only changed the docs. I don't know why the checks keep failing. It says it is having problems with ffmpeg?

Thank you so much for you help @stevhliu I think I should open a new PR. I made a mess here

I created a new pull request #23318 closing this PR
Compute the mask in-place, with less memory reads, and on CUDA on `XLNetLMHeadModel`
Author: lezcano
Pull request: yes

When working on TorchInductor, I realised that there was a part from `XLNetLMHeadModel` that was being compiled to CPU code.

This PR should allow to fuse this operation with other CUDA operations in `torch.compile`. It also should be faster on eager mode, as it has a this implementation has a lower foot-print.

If in-place operations are not allowed even in non-grad context, I still believe that doing ones + tril rather than a ones + tril + zeros + cat should be faster simply due to the number of memory reads/writes.

I tested that this code produces the same results for `0 <= qlen,mlen < 10` and `same_length in (True, False)`.

@ArthurZucker  @younesbelkada

Comments:
_The documentation is not available anymore as the PR was closed or merged._
Only add files with modification outside doc blocks
Author: ydshieh
Pull request: yes

# What does this PR do?

Add files for doctesting only when they have modifications outside docstrings.

(offline message from Sylvain)
> One small improvement I see is for the docstrings: for now the tests are launched on a file if we modify it, but I would only launch it if docstrings are modified (e.g. check the modifications are correct) to go faster. If changes in the code of a model file (for instance) trigger a doctest failure we will see it after. 

Comments:
_The documentation is not available anymore as the PR was closed or merged._
KeyError when setting ignore_mismatched_sizes=True  in Blip2ForConditionalGeneration
Author: Yangr116
Pull request: no

### System Info

### Issue
When setting ```ignore_mismatched_sizes=True``` in ```Blip2ForConditionalGeneration```,  a KeyError is raised. 



### Who can help?

_No response_

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

### Env

```
transformers==4.28.0.dev0
```

### Reproduction

* blip2-flan-t5-xl
```
from transformers import Blip2ForConditionalGeneration

model_name = "Salesforce/blip2-flan-t5-xl" 
model = Blip2ForConditionalGeneration.from_pretrained(model_name, ignore_mismatched_sizes=True)
```
the output is ```KeyError: 'language_model.decoder.block.0.layer.2.DenseReluDense.wi_1.weight'```

* blip2-opt-2.7b
```
from transformers import Blip2ForConditionalGeneration

model_name = "Salesforce/blip2-opt-2.7b" 
model = Blip2ForConditionalGeneration.from_pretrained(model_name, ignore_mismatched_sizes=True)
```
the output is ```KeyError: 'language_model.lm_head.weight'```

### Expected behavior

I would like to revised the input resolution, which needs to set ``` ignore_mismatched_sizes=True```.

Comments:
I fixed this possible issue in #22564, and I would like to know whether there are any other reasons.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
fixing a bug about gradient accumulation in codeparrot_training
Author: ArmelRandy
Pull request: yes

# What does this PR do?

The gradient accumulation was not pausing. In order to fix this issue, I modified the training loop in order to incorporate a better use of  `Accelerator` to handle gradient accumulation.

I also modified the declaration of accelerator  in order to incorporate the argument `gradient_accumulation_steps`.

To be tested.

Fixes #22541


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [x] Was this discussed/approved via a Github issue #22541 ? 
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

@lvwerra @loubnabnl 


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_22559). All of your documentation changes will be reflected on that endpoint.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
[docs] Fix Agents and Tools docstring
Author: stevhliu
Pull request: yes

Fixes the `kwarg` argument in the docstring to include what to expect, otherwise the `kwarg` gets mixed into the argument above it (see [here](https://huggingface.co/docs/transformers/main_classes/agent#transformers.Agent.chat.remote) for example).

Comments:
_The documentation is not available anymore as the PR was closed or merged._