noamaanMulla-03/transformers-issues huggingface dataset:
Was bert-large-uncased-whole-word-masking-finetuned-squad fine tuned or not.
Author: CaoZhongZ
Pull request: no

In the description of repo 'https://huggingface.co/bert-large-uncased-whole-word-masking-finetuned-squad', it states: "After pre-training, this model was fine-tuned on the SQuAD dataset with one of our fine-tuning scripts. See below for more information regarding this fine-tuning."

However I receive error messages when try to load the model using command:
```
model = AutoModelForQuestionAnswering.from_pretrained("bert-large-uncased-whole-word-masking-finetuned-squad")
```
It tells me I should fine tune it. So, should I or shouldn't I??

```
Some weights of BertForQuestionAnswering were not initialized from the model checkpoint at bert-large-uncased-whole-word-masking-finetuned-squad and are newly initialized: ['bert.encoder.layer.0.attention.self.distance_embedding.weight', 'bert.encoder.layer.1.attention.self.distance_embedding.weight', 'bert.encoder.layer.2.attention.self.distance_embedding.weight', 'bert.encoder.layer.3.attention.self.distance_embedding.weight', 'bert.encoder.layer.4.attention.self.distance_embedding.weight', 'bert.encoder.layer.5.attention.self.distance_embedding.weight', 'bert.encoder.layer.6.attention.self.distance_embedding.weight', 'bert.encoder.layer.7.attention.self.distance_embedding.weight', 'bert.encoder.layer.8.attention.self.distance_embedding.weight', 'bert.encoder.layer.9.attention.self.distance_embedding.weight', 'bert.encoder.layer.10.attention.self.distance_embedding.weight', 'bert.encoder.layer.11.attention.self.distance_embedding.weight', 'bert.encoder.layer.12.attention.self.distance_embedding.weight', 'bert.encoder.layer.13.attention.self.distance_embedding.weight', 'bert.encoder.layer.14.attention.self.distance_embedding.weight', 'bert.encoder.layer.15.attention.self.distance_embedding.weight', 'bert.encoder.layer.16.attention.self.distance_embedding.weight', 'bert.encoder.layer.17.attention.self.distance_embedding.weight', 'bert.encoder.layer.18.attention.self.distance_embedding.weight', 'bert.encoder.layer.19.attention.self.distance_embedding.weight', 'bert.encoder.layer.20.attention.self.distance_embedding.weight', 'bert.encoder.layer.21.attention.self.distance_embedding.weight', 'bert.encoder.layer.22.attention.self.distance_embedding.weight', 'bert.encoder.layer.23.attention.self.distance_embedding.weight']
You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.
```

Comments:
@CaoZhongZ which version are you using? I tried loading this model and it didn't throw any warning. 

Thanks for the hint! Might installed some spurious packages. A clean installation of transformers have no issue.
Use Bert model without pretrained weights
Author: avinashsai
Pull request: no

Hi,
I wanted to train a Bert classifier from scratch without any pretrained weights.  It has to be randomly initialized and trained. 

Example:
```
bert_base_model = BertForSequenceClassification()
trainer = Trainer(model=bert_base_model,
                  args=training_args,
                  train_dataset=train_loader,
                  eval_dataset=test_loader,
                  compute_metrics=compute_metrics
                  )
```

Comments:
If you want to train it from scratch you can use `BertConfig`, something like this:-

```python
from transformers import BertConfig, BertForSequenceClassification

config = BertConfig() # You can also change the architecture using this config class

model = BertForSequenceClassification(config=config)
```

Then you can proceed using the Trainer class to train the model.

thank you
I initially tried with this
```
bert_base_model = BertForSequenceClassification(BertConfig)
```
I was getting error.
accelerate scripts for question answering and qa with beam search
Author: theainerd
Pull request: yes

# What does this PR do?
Adding example script for question answering and question answering beam search using accelerate library . 

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors which may be interested in your PR.  @sgugger 

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @



Models:

- albert, bert, xlm: @LysandreJik
- blenderbot, bart, marian, pegasus, encoderdecoder,  t5: @patrickvonplaten, @patil-suraj
- longformer, reformer, transfoxl, xlnet: @patrickvonplaten
- fsmt: @stas00
- funnel: @sgugger
- gpt2: @patrickvonplaten, @LysandreJik
- rag: @patrickvonplaten, @lhoestq
- tensorflow: @LysandreJik

Library:

- benchmarks: @patrickvonplaten
- deepspeed: @stas00
- ray/raytune: @richardliaw, @amogkam
- text generation: @patrickvonplaten
- tokenizers: @n1t0, @LysandreJik
- trainer: @sgugger
- pipelines: @LysandreJik

Documentation: @sgugger

HF projects:

- datasets: [different repo](https://github.com/huggingface/datasets)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Examples:

- maintained examples (not research project or legacy): @sgugger, @patil-suraj
- research_projects/bert-loses-patience: @JetRunner
- research_projects/distillation: @VictorSanh

 -->


Comments:
Add a script to check inits are consistent
Author: sgugger
Pull request: yes

# What does this PR do?

Most inits in the project define the same objects twice (once in `_import_structure` and once in TYPE_CHECKING) to have a fast import so objects are only grabbed when actually needed. The problem is that those two halves have a tendency to diverge as contributors do not always pay attention to have them exactly match.

Well not anymore. Introducing `utils/check_inits.py`. This script will parse all the inits with two halves and return an error with a delightful and informative message telling the user what they did wrong. It is enforced in the CI and added to `make fixup` and `make quality`.

Comments:
fix incorrect case for s|Pretrained|PreTrained|
Author: stas00
Pull request: yes

This PR fixes incorrect `Pretrained` case for 2 cases:
```
git-replace PretrainedTokenizer PreTrainedTokenizer
git-replace transformers.PretrainedModel transformers.PreTrainedModel
```
there might be other cases to fix, but these stood out.

@sgugger 

Comments:
Documentation about loading a fast tokenizer within Transformers
Author: LysandreJik
Pull request: yes

This PR does two things:
- Allows to load a fast tokenizer from an instantiated `tokenizers` object
- Adds a page to document how to use these tokenizers within `transformers`

See [here](https://190138-155220641-gh.circle-artifacts.com/0/docs/_build/html/fast_tokenizers.html) for the generated docs

Comments:
Thanks for the suggestion @sgugger!
Add example for registering callbacks with trainers
Author: amalad
Pull request: yes

# What does this PR do?

Fixes the issue addressed in #9036 by adding an example for registering a custom callback with the Trainer.


## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

Fixes: https://github.com/huggingface/transformers/issues/9036

## Who can review?

Anyone in the community is free to review the PR. But @sgugger seems the most appropriate.

Comments:
Thanks for updating, this looks great!
[docs] missing info on call back registry
Author: stas00
Pull request: no
Labels: name: Good First Issue

https://huggingface.co/transformers/main_classes/callback.html is missing instructions/examples on how to register a callback. Thanks.

@sgugger 

Comments:
I guess the corresponding test demonstrates the usage: https://github.com/huggingface/transformers/blob/5c0bf39782c9eac8df55b89518f61c430862a7f6/tests/test_trainer_callback.py




I don't think this issue needs to be closed, one more example could be added to the documentatio! Let's make a good first issue out of it, maybe a contributor could help there :-)

Hi @sgugger. I'm new and I'd like to start contributing. Can I work on this issue?

Sure!

Thanks. Would an example like this be okay? - 
```
class MyCallback(TrainerCallback):
    "A callback that prints a message at the beginning of training"
    
    def on_train_begin(self, args, state, control, **kwargs):
        print("Starting training")

trainer = Trainer(
    model,
    args,
    train_dataset=train_dataset,
    eval_dataset=eval_dataset,
    callbacks=[MyCallback]
)
```
Also, should I add the example to https://huggingface.co/transformers/main_classes/callback.html, or would the [training tutorial](https://huggingface.co/transformers/training.html) be a better place?

I think is should also show how to use `add_callback` as an alternative too, otherwise, that's the gist of it. The callbacks page is perfect for this I think.

Thanks, I've added an example for `add_callback` too and opened a PR [here](https://github.com/huggingface/transformers/pull/10928). It's failing some tests right now but I'm not sure why since I've only modified the `callback.rst` file. Could you please help me figure out why this might be happening?

The CI is flakey at times, I restarted the failing jobs. It looks like network problems, may have to restart again later if it still fails.

edit: no luck, still network issues, will try again later, but do not worry, as long as the docs job passes and it does - you're good.

Cool, thanks!
Add `examples/language_modeling/run_clm_no_trainer.py`
Author: hemildesai
Pull request: yes

# What does this PR do?

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

This PR adds an example of finetuning a Causal Language Model (without using Trainer) to show the functionalities of the new accelerate library.

## Who can review?

@sgugger 

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- albert, bert, xlm: @LysandreJik
- blenderbot, bart, marian, pegasus, encoderdecoder,  t5: @patrickvonplaten, @patil-suraj
- longformer, reformer, transfoxl, xlnet: @patrickvonplaten
- fsmt: @stas00
- funnel: @sgugger
- gpt2: @patrickvonplaten, @LysandreJik
- rag: @patrickvonplaten, @lhoestq
- tensorflow: @LysandreJik

Library:

- benchmarks: @patrickvonplaten
- deepspeed: @stas00
- ray/raytune: @richardliaw, @amogkam
- text generation: @patrickvonplaten
- tokenizers: @n1t0, @LysandreJik
- trainer: @sgugger
- pipelines: @LysandreJik

Documentation: @sgugger

HF projects:

- datasets: [different repo](https://github.com/huggingface/datasets)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Examples:

- maintained examples (not research project or legacy): @sgugger, @patil-suraj
- research_projects/bert-loses-patience: @JetRunner
- research_projects/distillation: @VictorSanh

 -->


Comments:
@sgugger Awesome, I'll create a new PR for it.

Merging this one then, thanks again!
Add center_crop to ImageFeatureExtractionMixin
Author: sgugger
Pull request: yes

# What does this PR do?

This PR adds the center crop operation to `ImageFeatureExtractionMixin`. It works on PIL Images, NumPy arrays or torch tensors and behaves the same way as torchvision.transforms.CenterCrop: if the image is smaller than the desired size on a given dimension, it will be padded in that dimension.

Tests are added to check it works for all three types an give consistent results.

Comments:
Replace pkg_resources with importlib_metadata
Author: konstin
Pull request: yes

# What does this PR do?

Fixes #10964. The other reason for this change is that pkg_resources has been [deprecated](https://github.com/pypa/setuptools/commit/8fe85c22cee7fde5e6af571b30f864bad156a010) in favor of importlib_metadata.

`ImportError` looked like the best replacement for `pkg_resources.VersionConflict` since importlib has no equivalent exception, but I'm happy to change it to better suggestions.

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case. #10964
- [x] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation). n/a
- [x] Did you write any new necessary tests? n/a


## Who can review?

CC @stas00


Comments:
This is odd, CI is not being triggered 2nd time in a row!

I pushed an empty commit and now it started the CI.
Fix distributed gather for tuples of tensors of varying sizes
Author: sgugger
Pull request: yes

# What does this PR do?

Currently, the `DistributedTensorGatherer` responsible for gathering all predictions fails if the arrays to gather are a tuple of tensors, one of which has fixed dimensions (apart form the batch dimensions) and another varying sequence lengths. This is the case if the model has `output_hidden = True` in a sequence classification problem, for instance, as highlighted in #11055.

This PR reworks the internal of `DistributedTensorGatherer` so the storage is expanded when needed inside the recursive call instead of doing it recursively for all the tensors inside. It also adds a test that was failing before the PR.

Fixes #11055

Comments:
Make a base init in FeatureExtractionMixin
Author: sgugger
Pull request: yes

# What does this PR do?

This PR adds a base init to `FeatureExtractionMixin` so that high up in the hierarchy, remaining kwargs are set as attributes. Without this, adding new fields to the config of a feature extractor inheriting directly from `FeatureExtractionMixin`, like `ViTFeatureExtractor`, will fail so we won't be able to add fields (like the model type for easy use with an Auto API) without breaking backward compatibility.

Comments:
Add Readme for language modeling scripts with custom training loop and accelerate
Author: hemildesai
Pull request: yes

# What does this PR do?

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

Adds docs for `run_[mlm|clm]_no_trainer.py` scripts.

## Who can review?

@sgugger 

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- albert, bert, xlm: @LysandreJik
- blenderbot, bart, marian, pegasus, encoderdecoder,  t5: @patrickvonplaten, @patil-suraj
- longformer, reformer, transfoxl, xlnet: @patrickvonplaten
- fsmt: @stas00
- funnel: @sgugger
- gpt2: @patrickvonplaten, @LysandreJik
- rag: @patrickvonplaten, @lhoestq
- tensorflow: @LysandreJik

Library:

- benchmarks: @patrickvonplaten
- deepspeed: @stas00
- ray/raytune: @richardliaw, @amogkam
- text generation: @patrickvonplaten
- tokenizers: @n1t0, @LysandreJik
- trainer: @sgugger
- pipelines: @LysandreJik

Documentation: @sgugger

HF projects:

- datasets: [different repo](https://github.com/huggingface/datasets)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Examples:

- maintained examples (not research project or legacy): @sgugger, @patil-suraj
- research_projects/bert-loses-patience: @JetRunner
- research_projects/distillation: @VictorSanh

 -->


Comments:
using RAG with local documents
Author: saichandrapandraju
Pull request: no

Hi,
I have a requirement that model should search for relevant documents before answering the query. I found RAG from Facebook AI which perfectly aligns with my requirement . Also saw [this](https://huggingface.co/blog/ray-rag) blog from HF and came to know that HF implemented RAG which is awesome!

My doubt is whether I could extend HF's implementation of RAG so that the model should retrieve documents from local directory rather than HF's wikipedia corpus.
Are there any notebooks to refer..?

Comments:
duplicated
fix code quality issues
Author: withshubh
Pull request: yes

### Description

Hi :wave: I sent PR #8950 which has changes in a lot of files so sending this PR with fixes in few files so that it can be easily reviewed.

You can have a look at the various issues that were caught in the codebase [here](https://deepsource.io/gh/withshubh/transformers/issues/?category=recommended).

### Summary of changes

- Removed length check in favour of truthiness of the object
> Boosts minor performance, see the description [here](https://deepsource.io/gh/withshubh/transformers/issue/PYL-C1801/description/).
- Removed unnecessary comprehension
> boosts minor performance, see the description [here](https://deepsource.io/gh/withshubh/transformers/issue/PTC-W0016/description/).
- Removed unnecessary use of comprehension
> boosts minor performance, see the description [here](https://deepsource.io/gh/withshubh/transformers/issue/PTC-W0019/description/).
- Refactored the comparison involving `not`
> fixed [antipattern](https://deepsource.io/gh/withshubh/transformers/issue/PYL-C0113/description/)
- Removed unnecessary return statement
> removes [antipattern](https://deepsource.io/gh/withshubh/transformers/issue/PYL-R1711/description/)
- Iterated dictionary directly
> removes [antipattern](https://deepsource.io/gh/withshubh/transformers/issue/PYL-C0201/description/)
- Used literal syntax instead of function calls to create data structure
> boosts minor [performance](https://deepsource.io/gh/withshubh/transformers/issue/PYL-C1801/description/)
- Added .deepsource.toml
> config file to continuously analyze the repo for code quality issues

Comments:
Hi @LysandreJik :wave: Please review this PR

Hi @withshubh, as said before in https://github.com/huggingface/transformers/pull/8950#issuecomment-781222459, we would like to stay with our current tooling for now. Thank you.
HF emoji unicode doesn't work in console
Author: stas00
Pull request: yes

It doesn't look like using 🤗 is a great idea for printing to console. See attachment.

![snapshot_21](https://user-images.githubusercontent.com/10676103/113665185-e604cb00-9661-11eb-9b4b-d0f0423c6695.png)

This PR proposes to replace 🤗 with "HuggingFace" for an exception message.

@LysandreJik


Comments:
using RAG with local documents
Author: saichandrapandraju
Pull request: no

Hi,
I have a requirement that model should search for relevant documents to answer the query and I found [RAG](https://arxiv.org/pdf/2005.11401.pdf) from [Facebook AI](https://ai.facebook.com/) which perfectly fits my usecase. I also found [this](https://huggingface.co/blog/ray-rag) post in which HuggingFace explains RAG and came to know that HF implemented RAG which is awesome! 
My doubt is whether I could extend this functionality so that the model should do retrieval from local documents rather than from HF's wikipedia corpus.
Are there any notebooks to refer to? 

Comments:
Hello, thanks for opening an issue! We try to keep the github issues for bugs/feature requests.
Could you ask your question on the [forum](https://discusss.huggingface.co) instead?

Thanks!

ok. Created [issue](https://discuss.huggingface.co/t/using-rag-with-local-documents/5326) in [forum](https://discuss.huggingface.co)

Thanks @LysandreJik
added social thumbnail for docs
Author: philschmid
Pull request: yes

# What does this PR do?

I added OpenGraph/ Twitter Card support to the docs to create nice social thumbnails.

![Bildschirmfoto 2021-04-06 um 10 43 14](https://user-images.githubusercontent.com/32632186/113684071-25183800-96c5-11eb-8457-db42cc9931ae.png)

To be able to add these I needed to install `sphinxext-opengraph`. I came across this [issue](https://github.com/readthedocs/readthedocs.org/issues/1758) on the readthedocs repo saying that since someone has built this plugin they are not integrating and providing documentation to it. That's why I added it for creating the documentation. The repository can be found [here](https://github.com/wpilibsuite/sphinxext-opengraph/tree/main).

P.S. It seemed that `make style` never ran for `docs/` i hope the changes are okay otherwise I'll revert it. 

Comments:
Will do it tomorrow. Wanted to add it to `transformers` to be included in the next release