noamaanMulla-03/transformers-issues huggingface dataset:
Allow BatchEncoding to be pickled
Author: mfuntowicz
Pull request: yes

Overrides the methods ___get_state()___ & ___set_state()___ to (respectively) export the content of the underlying `data` dictionnary and - if defined - the content of `encodings`.

Unittests added to covert the serialization & deserialization of all the exported properties.
 
Block until new release of **tokenizers**

Comments:
Issues seems related to pretokenized input. Checking with @n1t0 what changed in-between 👍 

Indeed, I think we'll have to merge https://github.com/huggingface/transformers/pull/4510 before this.

Closing in favor of #5039
Allow pathlib.Path to be used on save_pretrained and save_vocabulary
Author: mfuntowicz
Pull request: yes
Labels: name: wontfix

Related to #4541

Signed-off-by: Morgan Funtowicz <morgan@huggingface.co>

Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/4650?src=pr&el=h1) Report
> Merging [#4650](https://codecov.io/gh/huggingface/transformers/pull/4650?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/e444648a302dc8520beec96356a4bf500944355c&el=desc) will **increase** coverage by `0.00%`.
> The diff coverage is `71.42%`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/4650/graphs/tree.svg?width=650&height=150&src=pr&token=9qOlN6Hb1c)](https://codecov.io/gh/huggingface/transformers/pull/4650?src=pr&el=tree)

```diff
@@           Coverage Diff           @@
##           master    #4650   +/-   ##
=======================================
  Coverage   77.42%   77.43%           
=======================================
  Files         128      128           
  Lines       21017    21019    +2     
=======================================
+ Hits        16273    16276    +3     
+ Misses       4744     4743    -1     
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/4650?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [src/transformers/tokenization\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/4650/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy90b2tlbml6YXRpb25fdXRpbHMucHk=) | `89.54% <71.42%> (+0.02%)` | :arrow_up: |
| [src/transformers/file\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/4650/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9maWxlX3V0aWxzLnB5) | `73.85% <0.00%> (+0.41%)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/4650?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/4650?src=pr&el=footer). Last update [e444648...d453872](https://codecov.io/gh/huggingface/transformers/pull/4650?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
[DOT NOT MERGE] Tokenizers Shape Polymorphism - Introduce pad_to_next_multiple_of parameters
Author: mfuntowicz
Pull request: yes

Needs new release of tokenizers (cc @n1t0) 

Comments:
That's a really cool feature. Looking forward to it!

Is the bucketization size not going to be too linear? Shouldn't we rather do `pad_to_next_power_of_two` or similar?

@julien-c That will be linear yet, by using a `power_of` growth we might rapidly increase the number of padding tokens to add and then fall into the opposite situation where most of the computation will be wasted on padding tokens.
[Prophetnet] Develop in parallel
Author: patrickvonplaten
Pull request: yes
Labels: name: model card

# What does this PR do?

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

Fixes # (issue)


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dimiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests), 
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to the it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests? 


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors which may be interested in your PR.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

 albert, bert, GPT2, XLM: @LysandreJik 
 tokenizers: @mfuntowicz
 Trainer: @sgugger
 Speed and Memory Benchmarks: @patrickvonplaten
 Model Cards: @julien-c
 Translation: @sshleifer
 Summarization: @sshleifer
 TextGeneration: @TevenLeScao 
 examples/distillation: @VictorSanh
 nlp datasets: [different repo](https://github.com/huggingface/nlp)
 rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)
 Text Generation: @TevenLeScao
 Blenderbot, Bart, Marian, Pegasus: @sshleifer
 T5: @patrickvonplaten
 Longformer/Reformer: @patrickvonplaten
 TransfoXL/XLNet: @TevenLeScao 
 examples/seq2seq: @sshleifer
 examples/bert-loses-patience: @JetRunner
 tensorflow: @jplu
 examples/token-classification: @stefan-it
 documentation: @sgugger
 -->

Comments:
Pre style
Author: sshleifer
Pull request: yes

# What does this PR do?

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

Fixes # (issue)


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to the it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors which may be interested in your PR.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

 albert, bert, XLM: @LysandreJik
 GPT2: @LysandreJik, @patrickvonplaten
 tokenizers: @mfuntowicz
 Trainer: @sgugger
 Benchmarks: @patrickvonplaten
 Model Cards: @julien-c
 Translation: @sshleifer
 Summarization: @sshleifer
 examples/distillation: @VictorSanh
 nlp datasets: [different repo](https://github.com/huggingface/nlp)
 rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)
 Text Generation: @patrickvonplaten, @TevenLeScao
 Blenderbot, Bart, Marian, Pegasus: @sshleifer
 T5: @patrickvonplaten
 Rag: @patrickvonplaten, @lhoestq
 EncoderDecoder: @patrickvonplaten
 Longformer, Reformer: @patrickvonplaten
 TransfoXL, XLNet: @TevenLeScao, @patrickvonplaten
 examples/seq2seq: @sshleifer
 examples/bert-loses-patience: @JetRunner
 tensorflow: @jplu
 examples/token-classification: @stefan-it
 documentation: @sgugger
 -->


Comments:
[summarization example] better error message
Author: stas00
Pull request: yes

https://github.com/huggingface/transformers/pull/15125 introduced a new required arg. 

This PR improves on the error message that is easier to act on as the original is not actionable unless one intimately knows the script.

@sgugger 

Comments:
Wait, this is an optional argument that only needs to be set if MBart is used as a model.

There is an assert later on in the script, so this test should just be removed from the postinit.

you're saying it was invalid logic in the first place in the original PR, correct?

Need more tests then since the issue wasn't caught by normal live tests?

There is a [test](https://github.com/huggingface/transformers/blob/1eb40338ac41a2ffbb2137216de3bb63bb739aad/examples/pytorch/test_examples.py#L350) that catches the problem, it's just marked as slow.

ah, right, I forgot about the slow ones! thank you, Sylvain.

So closing this as it is no longer relevant due to revert in https://github.com/huggingface/transformers/commit/96881729ce83cfc8e5fa04c903ee4296ad17cfbb
[wip] [ci] doc-job-skip take #4.5 dry-run via github direct edit
Author: stas00
Pull request: yes

This is take 4.5 on attempting to find a reliable way to get a list of modified files of this PR. It's identical to https://github.com/huggingface/transformers/pull/8980 but this PR was created from github UI direct file edit, so as we can see, it doesn't provide `CIRCLE_PR_NUMBER` - Nothing bad happens, but the check can't be done  since we have no information to work with :( 

It also happens with PR's made from a non-personal branch, https://github.com/huggingface/transformers/pull/9015
And the result is that the check is completely skipped as it has no data to work with:
https://app.circleci.com/pipelines/github/huggingface/transformers/17118/workflows/48285d78-cb04-4feb-87f8-77cb02ac2593/jobs/134493

Hoping that circlePR will fix that bug.

This PR:

* [x] tests a PR submission from non-personal forked repo
* [x] switches to `head.user.login` for the username to checkout the branch with - using PR username as it's in the master will not work if the branch is coming from a non-forked repo (original that is). (could also use `.head.repo.full_name` for the whole thing at once.)

For now I will let this PR sit for a while and add other fixes if we find more edge cases.


Comments:
Fix multiple deletions of the same files in save_pretrained
Author: sgugger
Pull request: yes

# What does this PR do?

Checkpoint sharding introduced a bug in `save_pretrained` for distributed setups, where the function is called on every process (TPU training for instance, or the scripts with Accelerate (see [this issue](https://github.com/huggingface/accelerate/issues/325)).
This changes the logic to only remove files when:
- they are different from existing ones (which should handle almost all cases since we can expect the save in a folder with existing weights to use the same model)
- we are on process 0.

Except `save_pretrained` does not know if we hare on process zero or not, so I use `save_config` to detect that. It looks like that argument was not aptly named and should be `is_main_process` instead. I can go ahead and deprecate/rename in this PR if you agree.

Comments:
_The documentation is not available anymore as the PR was closed or merged._
[WIP - Don't merge][EncoderDecoder] Extend Trainer for Bert2Bert
Author: patrickvonplaten
Pull request: yes
Labels: name: wontfix

This PR draft should be used to train a Bert2Bert model. 
It's not ready to be merged yet.

Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/5840?src=pr&el=h1) Report
> Merging [#5840](https://codecov.io/gh/huggingface/transformers/pull/5840?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/fa5423b1695cd24856bcff47214172e0f540d924&el=desc) will **decrease** coverage by `1.26%`.
> The diff coverage is `41.17%`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/5840/graphs/tree.svg?width=650&height=150&src=pr&token=9qOlN6Hb1c)](https://codecov.io/gh/huggingface/transformers/pull/5840?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master    #5840      +/-   ##
==========================================
- Coverage   77.79%   76.53%   -1.27%     
==========================================
  Files         145      145              
  Lines       25355    25365      +10     
==========================================
- Hits        19726    19414     -312     
- Misses       5629     5951     +322     
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/5840?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [src/transformers/trainer.py](https://codecov.io/gh/huggingface/transformers/pull/5840/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy90cmFpbmVyLnB5) | `37.41% <37.50%> (-0.55%)` | :arrow_down: |
| [src/transformers/training\_args.py](https://codecov.io/gh/huggingface/transformers/pull/5840/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy90cmFpbmluZ19hcmdzLnB5) | `77.77% <100.00%> (+0.22%)` | :arrow_up: |
| [src/transformers/modeling\_tf\_mobilebert.py](https://codecov.io/gh/huggingface/transformers/pull/5840/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ190Zl9tb2JpbGViZXJ0LnB5) | `23.38% <0.00%> (-73.39%)` | :arrow_down: |
| [src/transformers/modeling\_tf\_roberta.py](https://codecov.io/gh/huggingface/transformers/pull/5840/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ190Zl9yb2JlcnRhLnB5) | `93.36% <0.00%> (+49.37%)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/5840?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/5840?src=pr&el=footer). Last update [fa5423b...d9f6d07](https://codecov.io/gh/huggingface/transformers/pull/5840?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
[WIP RAG] Finalize RAG parallel
Author: patrickvonplaten
Pull request: yes

finalizing RAG in parallel to https://github.com/huggingface/transformers/pull/6813

Comments:
https://github.com/huggingface/transformers/blob/60c8defa01049b559933ad4e2ab3cc32c8b7ef27/src/transformers/tokenization_rag.py#L58

Hello, thank you for the great work! It seems like the first argument passed to the method, `self`, needs to be deleted.

Closing this PR as it served its purpose. Continue PR together with @ola13 here: https://github.com/huggingface/transformers/pull/6813
Fixup no_trainer save logic
Author: muellerzr
Pull request: yes
Labels: name: PyTorch

name: Examples

# Fix save logic in all `no_trainer` examples

## What does this add?

This PR fixes a bug pointed out in https://github.com/huggingface/accelerate/issues/322, where the save and load logic was wrong in how it skipped over the steps in the training loop.

This PR fixes it and changes the internals slightly to let saveing of a checkpoint be named right (before it always started at `epoch_0`, even if we resumed from epoch 1

Comments:
_The documentation is not available anymore as the PR was closed or merged._
Fix check_all_models_are_tested
Author: ydshieh
Pull request: yes

# What does this PR do?

The block from L396 to L398 should be in the `else` block if I understand correctly. 

https://github.com/huggingface/transformers/blob/8d3f952adb8c98cec2ea1f59bb7acfbc08232381/utils/check_repo.py#L394-L398

Otherwise, when a model has no test file, I get errors like below, and the program stops immediately.
(with `test_file = []` passed to `check_models_are_tested`)

```python
  File "/home/yih_dar_huggingface_co/transformers/utils/check_repo.py", line 362, in check_models_are_tested
    tested_models = find_tested_models(test_file)
  File "/home/yih_dar_huggingface_co/transformers/utils/check_repo.py", line 343, in find_tested_models
    with open(os.path.join(PATH_TO_TESTS, test_file), "r", encoding="utf-8", newline="\n") as f:
  File "/usr/lib/python3.9/posixpath.py", line 90, in join
    genericpath._check_arg_types('join', a, *p)
  File "/usr/lib/python3.9/genericpath.py", line 152, in _check_arg_types
    raise TypeError(f'{funcname}() argument must be str, bytes, or '
TypeError: join() argument must be str, bytes, or os.PathLike object, not 'list'
```

Comments:
_The documentation is not available anymore as the PR was closed or merged._
Add -e flag to some GH workflow yml files
Author: ydshieh
Pull request: yes

# What does this PR do?

Current 2 GitHub actions workflow yml use `pip install [.dev]`. This installs `transformers` in `/home/runner/venv/lib/python3.6/site-packages`, and this is cached. In future job runs, the cache is restored, and that `transformers` version is used - instead of the latest commit, i.e. we want to use `/home/runner/work/transformers/transformers/src`.

Without this PR, I have trouble after updating `add_new_model.py` (to change test model folders from `tests/` to `tests/models/`) because the `add_new_model.py` from `/home/runner/venv/lib/python3.6/site-packages` would be used, which would put the test template models under `tests/`

This PR makes sure the latest `transformers` is used by using `pip install -e [.dev]`, and builds a new cache with it.

- Add -e flag to some GH workflow yml files
- change cache key in order to make the change effective
- add a check on `transformers` location

Comments:
_The documentation is not available anymore as the PR was closed or merged._

The added check will produce something like (if the transformers is not from the expected location)

<img width="777" alt="Screenshot 2022-04-27 131320" src="https://user-images.githubusercontent.com/2521628/165506295-901fd17d-17f6-4fb7-8eec-f96b87d63b03.png">



after the new cache is built the first time and job completes (pass the check I added too), the next run when we have 

```
Cache restored from key: v3-tests_model_like-ce386d6c28d7afcca58dc875de2ef1b7477e8246a0bfdb6ff4de0eb222eafef2
```
the check failed with

```
transformers is from  but it shoud be from /home/runner/work/transformers/transformers/src.
A fix is required. Stop testing.
```
which means `pip show transformers` gives empty location!

I will try to make it work - I really like to have this test.
~~(But things should work now if we just remove this test)~~



Confirmed this (with `-e`) currently not working for the subsequentially run (i.e. cache loaded).

I still think there might be some workaround, let me try. Set to draft for now

### currently error
```
Traceback (most recent call last):
  File "/home/runner/venv/bin/transformers-cli", line 33, in <module>
    sys.exit(load_entry_point('transformers', 'console_scripts', 'transformers-cli')())
  File "/home/runner/venv/bin/transformers-cli", line [22](https://github.com/huggingface/transformers/runs/6197315688?check_suite_focus=true#step:7:22), in importlib_load_entry_point
    for entry_point in distribution(dist_name).entry_points
  File "/home/runner/venv/lib/python3.6/site-packages/importlib_metadata/__init__.py", line 815, in distribution
    return Distribution.from_name(distribution_name)
  File "/home/runner/venv/lib/python3.6/site-packages/importlib_metadata/__init__.py", line 430, in from_name
    raise PackageNotFoundError(name)
importlib_metadata.PackageNotFoundError: No package metadata was found for transformers
Error: Process completed with exit code 1.
```
[Research] Speed up evaluation for XTREME-S
Author: anton-l
Pull request: yes

# What does this PR do?

This adds a couple of improvements to the evaluation parts of the XTREME-S script:
* fix the bug where filtering by language happened multiple times for parallel workers (redundantly)
* use `preprocess_logits_for_metrics` to transform the logits into pred_ids before concatenating them to avoid OOMs
* add the `--language_group` parameter to train on the FLEURS dataset in batches of languages (west/eastern european languages, south asian languages etc.)

Misc:
* add `--ctc_zero_infinity` to handle the noisy FLEURS transcriptions

Comments:
_The documentation is not available anymore as the PR was closed or merged._

@patrickvonplaten these are ready to merge now I think 

Also cc @sanchit-gandhi: the fixes should make your life much easier if you decide to do a run of multilingual translation :)
Update check_models_are_tested to deal with Windows path
Author: ydshieh
Pull request: yes

# What does this PR do?

`TEST_FILES_WITH_NO_COMMON_TESTS` contains forward slash like `mt5/test_modeling_flax_mt5.py`.

The condition `if test_file in TEST_FILES_WITH_NO_COMMON_TESTS:` in `check_models_are_tested` would give failures to fix on Windows, like

```
camembert\test_modeling_camembert.py should define `all_model_classes` to apply common tests
```

This PR uses  `test_file.replace(os.sep, "/")` to make it work on Windows too 😄 

Comments:
_The documentation is not available anymore as the PR was closed or merged._
[T5 3B Covid 19] Adapt T5 TF conversion script to handle covid-19 3b t5
Author: patrickvonplaten
Pull request: yes
Labels: name: wontfix

This PR shows which changes were necessary to convert the 3B (and 11B) T5 model from this issue: https://github.com/huggingface/transformers/tree/adapt_t5_for_covid_19_3b to PyTorch. 

It might be possible that the official T5 library has changed in which case this code might be useful again. 
For now, this PR stays a draft though, but can be cleaned and merged if more T5 Conversion issues arise. 

Pinging @sshleifer @thomwolf for notification.

Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.