noamaanMulla-03/transformers-issues huggingface dataset:
Improve coverage of the documentation
Author: sgugger
Pull request: no
Labels: name: Documentation

name: Good First Issue

Currently, some public classes are not documented anywhere because we didn't create the corresponding doc pages. Those missing pages are:

- Benchmark classes
- Bert Japanese
- Data collators

If someone feels like working on one of those, please tag yourself with a comment on this issue. Once the objects are properly documented, they can be removed from the `SHOULD_BE_DOCUMENTED` constant in [this file](https://github.com/huggingface/transformers/blob/1310e1a758edc8e89ec363db76863c771fbeb1de/utils/check_repo.py#L374).


Comments:
I added docs for Bertweet [here](https://github.com/huggingface/transformers/pull/9379), first contribution, let me know if there is anything missing

Reopening the issue are not all of the items are fixed yet!

Hello @sgugger,

If no one else is working on it yet, I'd like to work on the `Bert Japanese` document.
(I also am interested in working on `Data collators`, but I’d like to do that one by one. If there is someone else who would like to work on, please give priority to that person.)



Go ahead @forest1988 :-)

Thanks, I'll do my best!

I deeply apologize for my delay in opening a PR for Bert Japanese.
I've just opened the PR.

https://github.com/huggingface/transformers/pull/11219

If you find any flaws, please let me know. I'll correct it soon.
Fix GPT-2 warnings
Author: LysandreJik
Pull request: yes

There was a forgotten code path when identifying missing weights.
When loading from a pytorch checkpoint to a tensorflow checkpoint, there was no issue, but doing so the other way around wouldn't check the `_keys_to_ignore_on_load_missing` and `_keys_to_ignore_on_load_unexpected` variables before printing a warning.

closes https://github.com/huggingface/transformers/issues/11192

Comments:
wav2vec2 converter: create the proper vocab.json while converting fairseq wav2vec2 finetuned model
Author: cceyda
Pull request: yes

# What does this PR do?
While converting a finetuned wav2vec2 model we also need to convert the related dictionary `dict.ltr.txt` to hugging face `vocab.json` format.

If a `dict_path` is specified:
  - Creates&saves the necessary vocab.json file  
  - Modifies config file special token ids and vocab size accordingly
  - Creates a processor with the right special tokens and saves the processor `preprocessor_config.json`

## Before submitting
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Did you make sure to update the documentation with your changes? -> not sure if there are any docs related to this.
- [ ] Did you write any new necessary tests? -> not sure if there are tests related to this.


## Who can review?

Models:
- wav2vec2: @patrickvonplaten @LysandreJik 



Comments:
Thanks for opening a PR @cceyda! I'll let Patrick review this one, but he's off for a couple of weeks. 

Maybe @patil-suraj can give it a look in the meantime, but we'll probably have to wait for Patrick to review as he has the most extensive experience with w2v2!

@LysandreJik  Merging this PR. The changes look good to me and do not affect the modeling file at all, so it won't cause any issues there.

Works for me! Thanks @cceyda and @patil-suraj
Add documentation for BertJapanese
Author: forest1988
Pull request: yes

# What does this PR do?

Add documentation for BertJapanese

Regarding #9035


## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Model: bert: @LysandreJik
Documentation: @sgugger

Comments:
Thanks a lot for your help!

@sgugger @LysandreJik 
Thank you for your quick reviewing this PR!

@sgugger 
Thank you for telling me how to format the code block!

> Thanks a lot for your PR! I made a couple of suggestions. Mostly, the example should follow the syntax
> 
> ```
> >>> code_line_1
> >>> code_line_2
> result
> ```

I've split the big code block into two, and then make all lines prefixed with `>>>`.
Now I think I can understand the format.
However, I wonder why BertTweet and BertGeneration, which I referred to before opening this PR, has code blocks without using `>>>` in them.
Are there any specific reasons? (Could it be because the output is not specifically described?)
Or, may I correct them using `>>>`?


We should always have the `>>>` as it allows us to use `doctest` which will test the example (it's been deactivated for a while but we will bring it back to life soon). So if you want to add those to some examples where it's missing, go ahead!

The only instance where we would not want those `>>>` is if we don't want the example to be tested.

Thanks for the detailed explanation about the prefix!
Now, I would like to add `>>>` to examples without the prefix, as far as I can find (except for which you don't want to be tested).
Rework examples/ to overwrite cache_dir for datasets too. 
Author: philschmid
Pull request: no

# 🚀 Feature request

Currently, you can pass [cache_dir](https://github.com/huggingface/transformers/blob/ef102c48865d70ff354b8ba1488d3fa8bfc116d8/examples/seq2seq/run_summarization.py#L79) into the `examples/` script to overwrite the `cache_dir` of [model config and tokenizers](https://github.com/huggingface/transformers/blob/ef102c48865d70ff354b8ba1488d3fa8bfc116d8/examples/seq2seq/run_summarization.py#L336). 
What would be the best way to adjust this to be able to use the `cache_dir` parameter for `datasets` [load_dataset](https://github.com/huggingface/transformers/blob/ef102c48865d70ff354b8ba1488d3fa8bfc116d8/examples/seq2seq/run_summarization.py#L313) method and `transformers`


## Motivation

When running training on Amazon SageMaker the cache_dir (`~/.cache/`) is not mounted to an EBS and cannot be increased. Therefore we need an option to point the `cache_dir` to an EBS backed directory for using the `examples/` scripts with large datasets. 


Comments:
Well someone would need to go other all the examples and add it as an argument in those calls to `load_dataset`.

Would you still keep it in 
```python
@dataclass
class ModelArguments:
```
I can adjust the examples in the near future. I just wanted to align on we can adjust this. 

We can switch it but to be honest it's more an internal class to the script than something of real significance so we don't really care.
Make sure code blocks are indented with four spaces
Author: sgugger
Pull request: yes

# What does this PR do?

In the documentation, many code blocks are indented with two or three spaces instead of 4. This PR enforces the use of 4 by:
- replacing indents of less than 4 by 4 in `make style`/`make fixup`
- checking there is no indent less than 4 in code blocks during `make quality`


Comments:
Ah, I don't match the `.. code-block:: xxx`, thanks for pointing that out!

OK, the next problem is the one I was concerned about in the first place (which is why i didn't offer a perl one liner). Your fixing code isn't shifting the whole block, but only the outside lines. so it now results in broken python code - bad indentation. e.g. look at any code blocks in the diff that start with `class`, or `{`

So when you shift whitespace - you have to do it for the whole code block.

That's too hard to fix and will result in other problems (what about bash commands that have different indents with the \?). So I will remove the automatic fix and just put a hard error the user will have to manually fix.

> That's too hard to fix and will result in other problems (what about bash commands that have different indents with the ?). 

Bash has no indentation issues. Only python does AFAIK. Perhaps you meant multiline python fed into a bash shell?

I think it's about finding the difference in the required whitespace, and applying this exact change to the whole block should make it work. Since everything will be shifted by the same number of characters, which is what the manual fix would do anyway.

> So I will remove the automatic fix and just put a hard error the user will have to manually fix.

Sure, that would work just fine.

Thank you.

Ok, I have something that can do the whole block, but since this PR already badly treated some parts, I need to go back from a fresh master, so closing this one.
[Deepspeed] zero3 tests band aid
Author: stas00
Pull request: yes

Currently Deepspeed integration is bleeding its global state which can impact other transformers runs in the same process w/o deepspeed, This only impacts tests that don't spawn a new process for deepspeed, which is not the norm.

This PR is just a temporary band-aid to restore the state on the test level - and I need to rethink of how to not use a global state or have the state tied to the deepspeed object which when destructed would automatically restore the state.

The main issue here is that the Trainer gets init'ed after the model, which is an important issue that I started the discussion on here:
https://github.com/huggingface/transformers/issues/10893

An example of the failing sequence is:
```
CUDA_VISIBLE_DEVICES=0 RUN_SLOW=1 pyt tests/deepspeed \
/test_deepspeed.py::TrainerIntegrationDeepSpeed::test_early_get_last_lr_1_zero3 tests/extended \
/test_trainer_ext.py::TestTrainerExt::test_run_seq2seq_no_dist
```



@LysandreJik

Comments:
Wav2Vec2 CommonVoice training - Save the processor before training starts
Author: Nithin-Holla
Pull request: yes

# What does this PR do?

Currently, the Wav2Vec2 processor is saved at the end of training. However, the vocabulary is non-deterministic and varies between runs. Thus, if the training is killed before it's done, the processor is not saved, meaning that the checkpoints do not contain the processor configuration files, making them unusable for resuming training or for evaluating on the checkpoint. Hence, this PR saves the processor before the training begins.

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

@patrickvonplaten @patil-suraj 


Comments:
Make "embeddings" plural in warning message within tokenization_utils_base
Author: jstremme
Pull request: yes

# What does this PR do?

Makes the word "embeddings" plural within the warning message in `tokenization_utils_base.py`.

## Before submitting
- [X] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [X] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


Comments:
Stale bot updated
Author: LysandreJik
Pull request: yes

This is an updated version of the stale bot.

**It is easier to review the file than the diff, you can find the file [here](https://github.com/huggingface/transformers/blob/d1e516ea0fe9e641a75a89e5a7522392f7dbd59d/scripts/stale.py).**

It sends a warning message after 23 days of inactivity, and closes the issue/PR if no activity is detected in the following 7 days.

It ignores the following labels (case insensitive):
- `Good First Issue`
- `Good Second Issue`
- `Feature Request`
- `New Model`
- `WIP`

If there are assignees on the issue/PR, then it puts the following comment: `f"This issue has been stale for a while, ping @{assignee.login}"`

I propose to leave the PR like it is, and I'll push an empty commit daily to check the result of the stale bot test (I'll remove other tests to ensure that we don't spend unnecessary CI credits). Once we verify that it works as expected for a few days, I propose to merge it.

You may check the results of the first run here: https://github.com/huggingface/transformers/runs/2045189559?check_suite_focus=true (Second commit was rate limited)

Comments:
@stas00, @sgugger, please review this PR. Here is a visualization of what would be done by the bot, were it to be merged today: https://github.com/huggingface/transformers/pull/10562/checks?check_run_id=2113061781

I have verified that all 11 issues that would be closed have received a warning 10 days ago. Thank you.
Close open files to suppress ResourceWarning
Author: parakalan
Pull request: yes

# Close open files to suppress ResourceWarning


<!--
Across the repo, we are opening a bunch of files and not closing them. This seems to cause issue when trying to programatically access them. This also raise ResourceWarning at a few places, for instance - transformers/convert_slow_tokenizer.py:308: ResourceWarning. This PR closes a few files which were left open after accessing.
-->


## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- albert, bert, xlm: @LysandreJik
- blenderbot, bart, marian, pegasus, encoderdecoder,  t5: @patrickvonplaten, @patil-suraj
- longformer, reformer, transfoxl, xlnet: @patrickvonplaten
- fsmt: @stas00
- funnel: @sgugger
- gpt2: @patrickvonplaten, @LysandreJik
- rag: @patrickvonplaten, @lhoestq
- tensorflow: @LysandreJik

Library:

- benchmarks: @patrickvonplaten
- deepspeed: @stas00
- ray/raytune: @richardliaw, @amogkam
- text generation: @patrickvonplaten
- tokenizers: @n1t0, @LysandreJik
- trainer: @sgugger
- pipelines: @LysandreJik

Documentation: @sgugger

HF projects:

- datasets: [different repo](https://github.com/huggingface/datasets)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Examples:

- maintained examples (not research project or legacy): @sgugger, @patil-suraj
- research_projects/bert-loses-patience: @JetRunner
- research_projects/distillation: @VictorSanh

 -->


Comments:
Fix dimention misspellings.
Author: odellus
Pull request: yes

# What does this PR do?

Replaces dimention mispelling with the proper spelling dimension. 


## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- albert, bert, xlm: @LysandreJik
- blenderbot, bart, marian, pegasus, encoderdecoder,  t5: @patrickvonplaten, @patil-suraj
- longformer, reformer, transfoxl, xlnet: @patrickvonplaten
- fsmt: @stas00
- funnel: @sgugger
- gpt2: @patrickvonplaten, @LysandreJik
- rag: @patrickvonplaten, @lhoestq
- tensorflow: @LysandreJik

Library:

- benchmarks: @patrickvonplaten
- deepspeed: @stas00
- ray/raytune: @richardliaw, @amogkam
- text generation: @patrickvonplaten
- tokenizers: @n1t0, @LysandreJik
- trainer: @sgugger
- pipelines: @LysandreJik

Documentation: @sgugger

HF projects:

- datasets: [different repo](https://github.com/huggingface/datasets)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Examples:

- maintained examples (not research project or legacy): @sgugger, @patil-suraj
- research_projects/bert-loses-patience: @JetRunner
- research_projects/distillation: @VictorSanh

 -->


Comments:
Weights used for Masked LM predictions
Author: simran-khanuja
Pull request: no

I wanted to get masked word predictions for a few bert-base models. I am converting the pytorch models to the original bert tf format using [this](https://github.com/huggingface/transformers/blob/master/src/transformers/models/bert/convert_bert_pytorch_checkpoint_to_original_tf.py) by modifying the code to load BertForPreTraining state_dict. I am unaware of the use of cls/predictions/decoder in the snippet below, to make the masked predictions. The original BERT codebase does not have this layer, hence. Is it used, or can I safely disregard this to obtain predictions?
![image](https://user-images.githubusercontent.com/24687672/104811962-01132c00-5825-11eb-81ef-c7f36c07f457.png)


Comments:
The `cls/predictions/decoder` is the linear layer that is used to project the output of the transformer to the vocabulary logits. This layer is *tied* to the input embeddings: it has the same weights.

I believe the original BERT codebase doesn't have this layer because it re-uses the input embedding layer, instead of instantiating another one. We do this too in the TF implementation.

You can, therefore, safely disregard this layer is the implementation you're using uses the input embeddings' weights to project the output of the transformer to the vocabulary logits. 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.
[troubleshooting] add 2 points of reference to the offline mode
Author: stas00
Pull request: yes

As discussed at https://github.com/huggingface/transformers/issues/11231#issuecomment-818976986 the offline mode doc can be hard to find, so this PR:

- starts a new "troubleshooting" document 
- adds a note xref to `from_pretrained`

Surely we can start populating the new "troubleshooting" document - the idea here is to have common problems with explicit error messages and pointers to solutions.

@LysandreJik, @sgugger  

Comments:
Bug in numpy_pad_and_concatenate
Author: liamcli
Pull request: no

https://github.com/huggingface/transformers/blob/77b862847b8069d57c0849ca012f48414c427d8e/src/transformers/trainer_pt_utils.py#L71

I believe this should be 
`np.concatenate((array1, array2), axis=0)`

Comments:
Indeed. Would you like to do a PR to fix this since you spotted the bug?

sure thing!

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.