noamaanMulla-03/transformers-issues huggingface dataset:
Pin rouge_score
Author: albertvillanova
Pull request: yes

# What does this PR do?

Temporarily pin `rouge_score` (to avoid latest version 0.7.0) until the issue is fixed on their side:
- https://github.com/google-research/google-research/issues/1212

See:
- https://github.com/huggingface/datasets/issues/4734


Comments:
_The documentation is not available anymore as the PR was closed or merged._

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

Hey @albertvillanova, sorry for missing this! Should this be merged? Should we replace by `!=0.7.0` so that we're still compatible with newer versions?

They made several failed releases until they got to fix the error... Let me update this PR with all the versions to be avoided.

@LysandreJik there is a non-passing test though...

Hmmm weird, it seems like it passed when it was <0.07 [here](https://github.com/huggingface/transformers/runs/7464301285?check_suite_focus=true) (installing version v0.0.4), but not after setting `rouge-score!=0.0.7,!=0.0.8,!=0.1,!=0.1.1` (installing version 1.2.0).

I see that when installing `rouge-score` version v1.2.0, it was installing `rouge-score` using a legacy approach:
```
Using legacy 'setup.py install' for rouge-score, since package 'wheel' is not installed.
```

Could this be the cause of the failure? If you put `<0.7` once again, does it pass the test?

The test does not pass now either with `rouge-score<0.0.7`, @LysandreJik. 

But it passed when I opened this PR: see https://github.com/huggingface/transformers/pull/18247/commits/4e7a46dcfe4bfbab25858052017a39459c7831d4
Fix minor typo in prose of model outputs documentation
Author: pcuenca
Pull request: yes

# What does this PR do?

Fixes a very minor typo in the documentation of the model outputs section.

## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Documentation: @sgugger


Comments:
_The documentation is not available anymore as the PR was closed or merged._
reflect max_new_tokens in `Seq2SeqTrainer`
Author: kumapo
Pull request: yes

# What does this PR do?

in most cases, VisionEncoderDecoderModel's `max_length` is set implicitly.
it leads to the problem if the model generates prediction given `max_new_tokens`.

this PR makes `max_new_tokens` handled as expected in `Seq2SeqTrainer. prediction_step()` in the case.

Fixes #18785

P.S. I can reproduce the issue if using `huggingface/transformers`.
but, using this PR with same codes to reproduce, no exceptions raised.

## Before submitting

- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

- trainer: @sgugger


Comments:
_The documentation is not available anymore as the PR was closed or merged._

@ydshieh, could you take a look at this when you have some time please? Thanks a lot!

Hi, @kumapo 

I believe it also requires a change in

https://github.com/huggingface/transformers/blob/main/src/transformers/trainer_seq2seq.py#L129

right?

@ydshieh, yes. at same time I believe `Seq2SeqTrainer.evaluate()` needs the same change.


@sgugger, thank you for your feedback. I've updated the PR.

It seems there is an issue with your CircleCI permissions, the tests won't run.
Could you try refreshing your permissions as shown [here](https://support.circleci.com/hc/en-us/articles/360048210711-How-to-Refresh-User-Permissions-)?

@LysandreJik, I've done all steps to refresh circleci permission.
but it seems that nothing happens with tests. let me know if I missed something to be known.

Can you try pushing an empty commit on your branch to re-trigger the tests?
```
git commit -m "Trigger CI" --allow-empty
```

To pass the test, you can run 

```bash
make style
```
and commit the change.
Run_mlm.py for fine-tuning generator(mlm) of electra 
Author: ToanKGO
Pull request: no

Hi, I have a new corpus and want to fine-tune ELECTRA for better result. ELECTRA used generator as generator after that used discriminator. Can run file run_mlm.py makes my pre-trained electra works better ?

Comments:
Hi @ToanKGO,

I think the latest update on the ELECTRA pretraining/fine-tuning is this comment from @LysandreJik :+1: 
https://github.com/huggingface/transformers/pull/4656#issuecomment-711082850

So you could use e.g. this implementation: https://github.com/richarddwang/electra_pytorch

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Pin revision for LayoutLMForQuestionAnswering and TFLayoutLMForQuestionAnswering tests
Author: ankrgyl
Pull request: yes

# What does this PR do?

The newly introduced tests for `LayoutLMForQuestionAnswering` and `TFLayoutLMForQuestionAnswering` broke due to a change to the weights in https://huggingface.co/impira/layoutlm-document-qa. To make sure a weights change does not break tests, I've pinned the revisions in these tests. I'll separately investigate the weights and debug if something is broken with them.

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.

It was raised here: https://github.com/huggingface/transformers/pull/18407.

- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

@NielsRogge @ydshieh

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- albert, bert, xlm: @LysandreJik
- blenderbot, bart, marian, pegasus, encoderdecoder,  t5: @patrickvonplaten, @patil-suraj
- longformer, reformer, transfoxl, xlnet: @patrickvonplaten
- fsmt: @stas00
- funnel: @sgugger
- gpt2: @patrickvonplaten, @LysandreJik
- rag: @patrickvonplaten, @lhoestq
- tensorflow: @LysandreJik

Library:

- benchmarks: @patrickvonplaten
- deepspeed: @stas00
- ray/raytune: @richardliaw, @amogkam
- text generation: @patrickvonplaten
- tokenizers: @n1t0, @LysandreJik
- trainer: @sgugger
- pipelines: @LysandreJik

Documentation: @sgugger

HF projects:

- datasets: [different repo](https://github.com/huggingface/datasets)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Examples:

- maintained examples (not research project or legacy): @sgugger, @patil-suraj
- research_projects/bert-loses-patience: @JetRunner
- research_projects/distillation: @VictorSanh

 -->


Comments:
_The documentation is not available anymore as the PR was closed or merged._
Add Image To Text Generation pipeline
Author: OlivierDehaene
Pull request: yes

# What does this PR do?

Add Image To Text Generation pipeline. The pipeline currently defaults to [nlpconnect/vit-gpt2-image-captioning](https://huggingface.co/nlpconnect/vit-gpt2-image-captioning).

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [x] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [x] Did you write any new necessary tests?


## Who can review?

This features was asked for by @Narsil.


Comments:
_The documentation is not available anymore as the PR was closed or merged._

cc @mishig25, the inference widgets for models like TrOCR, Donut, [image captioning](https://huggingface.co/nlpconnect/vit-gpt2-image-captioning) can now be created! 🥳
Learning rate is given as tensor, cannot serialize TrainerState in order to save checkpoint.
Author: kmckiern
Pull request: no
Labels: name: bug

### System Info

- `transformers` version: 4.21.1
- Platform: Linux-5.15.0-46-generic-x86_64-with-glibc2.35
- Python version: 3.10.5
- Huggingface_hub version: 0.8.1
- PyTorch version (GPU?): 1.12.1.post200 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: yes
- Using distributed or parallel set-up in script?: no

### Who can help?

@sgugger

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

1. use Adafactor optimizer and schedule
```
optimizer = Adafactor(
    model.parameters(),
    relative_step=True,
    warmup_init=True,
)
lr_scheduler = AdafactorSchedule(optimizer)
```

2. save a checkpoint

### Expected behavior

When using an `AdafactorSchedule` I can't use the `Trainer` class to save a checkpoint. 

It breaks [here](https://github.com/huggingface/transformers/blob/main/src/transformers/trainer_callback.py#L97) since the learning rate attached to the `TrainerState` is given by a tensor and tensors are not JSON serializable. 

I dropped a breakpoint at this line and took a look at my `TrainerState`: 
```
In [4]: self.log_history[0]['learning_rate']
Out[4]: tensor(0.0001, device='cuda:0')
```

Expected behavior is that the learning rate attached to the log history would be given by a float and would therefore be JSON serializable.

Comments:
Why does setting `--fp16 True` not save memory
Author: skye95git
Pull request: no

https://github.com/huggingface/transformers/blob/e54a1b49aa6268c484625c6374f952f318914743/examples/pytorch/language-modeling/run_mlm.py#L247-L250

https://github.com/huggingface/transformers/blob/e54a1b49aa6268c484625c6374f952f318914743/src/transformers/training_args.py#L663-L666

I want to pre-train Roberta on my dataset. However, the Batch size can be set to 32 at most. Otherwise, OOM is reported. I plan to use `Mixed-precision` to save memory. So I set `--fp16 True`. However, the Batch size can only be set to 32 at most. Otherwise, OOM will be reported. It seems that setting up `FP16` is not doing much to save memory.

Comments:
Hello, thanks for opening an issue! We try to keep the github issues for bugs/feature requests.
Could you ask your question on the [forum](https://discuss.huggingface.co) instead?

Thanks!

This question is now on the forum here: https://discuss.huggingface.co/t/why-does-setting-fp16-true-not-save-memory-as-expected/22400
Fix naming issue with Image2TextGenerationPipeline
Author: OlivierDehaene
Pull request: yes

# What does this PR do?

Fixes naming issue with the Image2TextGenerationPipeline: naming was not consistent with other libraries.
See [this message](https://huggingface.slack.com/archives/C014N4749J9/p1662091141349169?thread_ts=1662048034.910319&cid=C014N4749J9). 

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [x] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [x] Did you write any new necessary tests?


## Who can review?

@Narsil, @sgugger 


Comments:
_The documentation is not available anymore as the PR was closed or merged._
Clean up utils.hub using the latest from hf_hub
Author: sgugger
Pull request: yes

# What does this PR do?

This PR uses the newly released version of `hugginface_hub` to clean up a few things introduced in #18438 (points 1 and 2 in the description of this PR). For point 3 (load from cache) there is currently a difference between Transformers' `try_to_load_from_cache` and the huggingface_hub's one, so this will a follow-up PR in `huggingface_hub` (and then to wait for the next release).

Comments:
_The documentation is not available anymore as the PR was closed or merged._
IterableDatasets result in nan loss in eval with dataloader_num_workers>=1 and multi-gpu
Author: dlwh
Pull request: no
Labels: name: bug

### System Info

- `transformers` version: 4.22.0.dev0
- Platform: Linux-5.4.0-105-generic-x86_64-with-glibc2.31
- Python version: 3.9.13
- Huggingface_hub version: 0.8.1
- PyTorch version (GPU?): 1.12.1 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: YES


### Who can help?

@sgugger 

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

Run this modified/minimized [run_clm.py](https://gist.github.com/dlwh/074e2571fab15f94103603674dd184a3) under DeepSpeed (or presumably any other multiprocessing, but I didn't check)

The script works fine if you don't use multiprocessing, or if you change it to not use an IterableDataset, or if you set dataloader_num_workers to 0 (which is the default)

Relevant bit of logs:

```
Traceback (most recent call last):
  File "run_clm.py", line 125, in <module>
    main()
  File "run_clm.py", line 116, in main
    assert np.isfinite(metrics["eval_loss"])
AssertionError
```


### Expected behavior

assertion shouldn't fail, or at least trainer should require that dataloader_num_workers is 0 if using multi-gpu and IterableDataset...

The underlying issue is that Trainer creates 'IterableDatasetShard's when using multi-gpu and IterableDataset, and   (evaluation_loop)[https://github.com/huggingface/transformers/blob/main/src/transformers/trainer.py#L3024-L3027] looks at the "num_examples" property of the IterableDatasetShard, but this value isn't actually incremented in the main training process if you're using `dataloader_num_workers>0`, because it's set in the worker processes...

I will note that `evaluation_loop` goes to some trouble [to track the actual number of examples](https://github.com/huggingface/transformers/blob/main/src/transformers/trainer.py#L2935-L2944) so unless I'm missing something I think one could just always use that.

Comments:
Thanks for flagging. The PR above should fix the issue, could you give it a quick try?
pin Slack SDK to 3.18.1 to avoid failing issue
Author: ydshieh
Pull request: yes

# What does this PR do?

Currently CI Slack reports failed to be sent due to an error
```bash
The server responded with: {'ok': False, 'error': 'invalid_blocks_format'}
```
It happens since `slack-sdk-3.18.2`. This PR pin `slack-sdk-3.18.1` so we can receive the reports.

Comments:
_The documentation is not available anymore as the PR was closed or merged._
Fix number of examples for iterable datasets in multiprocessing
Author: sgugger
Pull request: yes

# What does this PR do?

As pointed out in #18608, `IterableDatasetShard.num_examples` is not always update in multiprocessing environments. This PR addresses that by ignoring the value in those cases. It also adds a stronger check of trusting the observed number of examples when for the reason the length is 0.

Fixes #18608

Comments:
_The documentation is not available anymore as the PR was closed or merged._
Generate: validate `model_kwargs` on TF (and catch typos in generate arguments)
Author: gante
Pull request: yes

# What does this PR do?

TF version of https://github.com/huggingface/transformers/pull/18261

Adds `model_kwargs` validation to TF `generate`, which also catches typos in the arguments. See the PR above for more details and an example of the error message users will see.

Since TF had no dedicated file for `generate` tests, I took the liberty to create it and move some existing tests there (>70% of the diff is due to moving things around :) ). The test for this new check was also added there. 

Comments:
_The documentation is not available anymore as the PR was closed or merged._
A script to download artifacts and perform CI error statistics
Author: ydshieh
Pull request: yes

# What does this PR do?

This script is helpful for the past CI project. It downloads all artifacts from a workflow run, and get the error statistics + the corresponding failing tests.

`errors.json`: the places where error occur + what those errors are

`failed_tests.json`: which test methods failed (can be used to determine which models will be supported in a specific backend version)

** We might adjust this  script a bit once we start to perform the automation of the past CI project. **

Currently, it prints something as the following (but we save the full information in 2 json files)
```bash
('RuntimeError: Integer division of tensors using div or / is no longer supported, and in a future release div will perform true division as in Python 3. Use true_divide or floor_divide (// in Python) instead.', 66)
("AttributeError: module 'torch.jit' has no attribute '_state'", 51)
("AttributeError: module 'torch' has no attribute 'minimum'", 45)
("AttributeError: module 'torch' has no attribute 'multiply'", 25)
('AttributeError: Caught AttributeError in replica 0 on device 0.', 3)
('RuntimeError: "normal_kernel_cpu" not implemented for \'BFloat16\'', 3)
("AssertionError: Couldn't trace module.", 3)
("AttributeError: module 'torch' has no attribute 'isneginf'", 2)
("TypeError: where(): argument 'input' (position 2) must be Tensor, not int", 2)
("AttributeError: 'Tensor' object has no attribute 'nansum'", 1)
('RuntimeError: Caught RuntimeError in replica 0 on device 0.', 1)
```



Comments:
_The documentation is not available anymore as the PR was closed or merged._
Remove cached torch_extensions on CI runners
Author: ydshieh
Pull request: yes

# What does this PR do?

- The test
  ```
  tests/deepspeed/test_deepspeed.py::TrainerIntegrationDeepSpeed::test_hf_scheduler_ds_optimizer
  ```
  failed since 2 weeks due to some cache issue. The error message is
  ```bash
  E   ImportError: /github/home/.cache/torch_extensions/py38_cu113/fused_adam/fused_adam.so: undefined symbol: _ZN3c104impl8GPUTrace13gpuTraceStateE`
  ```

- After I remove the cache (on the host runners, not inside the running docker) by
  ```bash
  sudo rm -rf /home/github_actions/actions-runner/_work_temp/_github_home/.cache/torch_extensions/py38_cu113/
  ```
  the test passes.

- This PR add the following in the workflow file
  ```bash
  rm -rf /github/home/.cache/torch_extensions/
  ```
  to avoid the same problem occurring in the future.

Remark: Notice the host directory
```
/home/github_actions/actions-runner/_work_temp/_github_home/
```
is mapped to 
```
 /github/home/
```
inside the running docker (we can see this in the job run page).

Comments:
_The documentation is not available anymore as the PR was closed or merged._
Update docs landing page
Author: stevhliu
Pull request: yes
Labels: name: Documentation

This PR updates the docs landing page to better describe what `transformers` is, what it offers, and briefly introduce users to its design. I think this gives a clearer picture of `transformers` and is more impactful than listing all the different tasks supported. Let me know what you think!

There's also a minor issue with the image for custom support. Nils is no longer with us, so we may want to update this image with another member of the team. No big deal though :)

Comments:
_The documentation is not available anymore as the PR was closed or merged._
Finetune guide for semantic segmentation
Author: stevhliu
Pull request: yes
Labels: name: Documentation

This PR creates a finetune guide for semantic segmentation in the docs. Unlike previous finetune guides, this one will include:

* metrics for evaluation
* a section for how to use the model for inference
* an embedded Gradio demo

🚧 To do:
- [x] create section for inference
- [ ] create Gradio demo

Comments:
_The documentation is not available anymore as the PR was closed or merged._