noamaanMulla-03/transformers-issues huggingface dataset:
Add Definition of a transformer to the glossary
Author: darigovresearch
Pull request: no

Thought it may be helpful to have an easy to understand definition for what a transformer is in the [glossary](https://huggingface.co/transformers/glossary.html) for any new joiners.

@sgugger any thoughts?

Happy to add a definition if provided with one in the open pull request #8949

Comments:
Are you looking for https://huggingface.co/transformers/model_summary.html ?

@cronoik thanks for taking a look! Wasn't looking for that page, was trying to find just a general definition of what a transformer is in terms of the general concept

Do you have something like this in mind: https://github.com/huggingface/transformers/blob/master/notebooks/02-transformers.ipynb ?

No, wasn't looking for a notebook, just a one line description/explanation of what a transformer is

How would you describe it to someone who doesn't know about transformers?

Maybe simply as self-attention based deep learning model architecture.

@cronoik That's a good start, `self-attention` & `deep learning` aren't yet defined in the glossary

How would you define those?

`self-attention`: each element of the input finds out which other elements of the input they should attend to.
`deep learning`: machine learning algorithms which uses NN which several layers.

@darigovresearch 

@cronoik thanks for that! Would you like to put in a pull request so that your definitions go into the transformers glossary and the set of flashcards that we built or would you like us to do it?

I'm sure those definitions would be welcome and easily merged by the maintainers

https://www.darigovresearch.com/huggingface-transformers-glossary-flashcards

Thanks @sgugger for merging the pull request, @cronoik your definitions are now on the glossary page and I have also added them to the flashcards so this issue can now be closed. Thank you both for your help!

Glossary https://huggingface.co/transformers/glossary.html

Flashcards https://www.darigovresearch.com/huggingface-transformers-glossary-flashcards

Thanks for your commits ;).
feat(wandb): logging and configuration improvements
Author: borisdayma
Pull request: yes

# What does this PR do?

Following improvements to `wandb` integration:
* ensure unique artifact id → previously it was based on run name which could create duplicates and mismatches (as runs can be updated manually in the UI)
* allow manual calls to `wandb.init()` → previously it would have closed the run and started a new one
* when a wandb run already exists (manually created), adds automatically model config parameters
* simplify reinit logic → now explicitly closes a run for hp search and avoid use of reinit which can have complex side effects (different behavior in notebooks vs scripts)
* ensure we have no dropped values (when step is below a previous logged value) by logging the step as an independent `train/global_step` metric and set it as default x-axis in the UI (can be edited manually). Note: this auto-setting of x-axis will be activated in upcoming release of wandb
* get values committed immediately so they appear in the UI with no delay
* fixes compatibility with sagemaker

Fixes https://github.com/wandb/client/issues/1499, #8754, #10486


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [X] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [X] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors which may be interested in your PR.

Documentation: @sgugger

Comments:
Fix p_mask cls token masking in question-answering pipeline
Author: mmaslankowska-neurosys
Pull request: yes

# What does this PR do?
It fixes a really small bug described in detail in the issue - it only adds a condition to the if statement responsible for unmasking the `cls_token_id` in the `p_mask` used in the question answering pipeline.

Fixes #10810 

## Who can review?
Anyone in the community is free to review the PR once the tests have passed. 

Comments:
[trainer] Fixes Typo in Predict Method of Trainer
Author: bhadreshpsavani
Pull request: yes

# What does this PR do?

Fixes typo in Predict Method of Trainer. This will enable saving files with the correct prefix `test` in Predict stage.  Earlier it was saving it with the `eval` prefix for both predict and evaluate stage.

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? This is discussed in #10482
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?:
@stas00  @sgugger 

Comments:
[Examples] Added predict stage and Updated Example Template
Author: bhadreshpsavani
Pull request: yes

# What does this PR do?
* Adds Predict stage in `run_xlni.py` text-classification example
* Updated Example Template for Predict Stage

Fixes #10482 

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? #10482
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?
@stas00 @sgugger 

Comments:
Sm trainer smp init fix
Author: philschmid
Pull request: yes

# What does this PR do?

Fixes `SageMakerTrainer` `smp.init` for `smp 1.3`. It also removes the `is_smdistributed_available` for are more robust `is_sagemaker_model_parallel_available`. 


Comments:
LGTM! I ran one training job successfully with these changes. 

I tested it with `pytorch1.7.1`
`564829616587.dkr.ecr.us-east-1.amazonaws.com/huggingface-pytorch-training:1.7.1-transformers4.4.0-py36-gpu-cu110-ubuntu18.04` 
and `pytorch1.6` 
`763104351884.dkr.ecr.us-east-1.amazonaws.com/huggingface-pytorch-training:1.6.0-transformers4.4.2-gpu-py36-cu110-ubuntu18.04`
Add new notebook links in the docs
Author: sgugger
Pull request: yes

# What does this PR do?

This PR adds links to the three missing tasks in the notebooks page: multiple choice, translation and summarization.

Comments:
Oh actually, an additional comment: the title of the summarization notebook is currently "Text classification on GLUE" (and same for the translation)

Fixed the titles and the sentence, so merging. Thanks for the review!
[Community notebooks] Add notebook for fine-tuning Bart with Trainer in two langs
Author: elsanns
Pull request: yes

Add a community notebook on fine-tuning Bart for summarization on wiki_lingua with Trainer. 
Includes:
- a non-English example (English, French)
- DataCollatorForSeq2Seq
- label padding with -100 (ignore in loss)
- Wandb integration

Comments:
Fix overflowing bad word ids
Author: LysandreJik
Pull request: yes

As of now, bad word IDs are not checked when added to the configuration/passed as inputs to the generate method.

This is an issue when an invalid bad word ID is defined: if the vocab size is 30k, then defining a bad word ID for `30001` crashes the generation function with the following error:
```
    torch.sparse.LongTensor(banned_mask.t(), indices, scores.size()).to(scores.device).to_dense().bool()
RuntimeError: size is inconsistent with indices: for dim 1, size is 30000 but found index 30001
```

Please let me know if you think this should raise a better error instead, rather than a warning.

Comments:
Remove version warning in pretrained BART models
Author: sgugger
Pull request: yes

# What does this PR do?

This PR fixes the warnings when loading any pretrained BART model:
```
Some weights of the model checkpoint at facebook/bart-large-mnli were not used when initializing BartModelForSequenceClassification: ['model.encoder.version', 'model.decoder.version']
- This IS expected if you are initializing BartModelForSequenceClassification from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).
- This IS NOT expected if you are initializing BartModelForSequenceClassification from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).
```

Comments:
Update Training Arguments Documentation: ignore_skip_data -> ignore_data_skip
Author: siddk
Pull request: yes

Currently, docs/docstring for TrainingArguments refers to `ignore_skip_data` as the argument for skipping dataloader replay on resume. However, actual argument is called `ignored_data_skip` which leads to errors if you just go off the docs.

(Separate note, doing a full replay for long runs is pretty annoying --> thinking about a way to eliminate this/speed up considerably, but would love to here what the Transformers team is up to in this regard!).

## Before submitting
- [X] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).

## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors which may be interested in your PR.

Tagging @sgugger as this has to do with documentation.

Comments:
Error while loading model file - .ckpt file :: Missing key(s) in state_dict
Author: adithyaan-creator
Pull request: no

@julien-c @patrickvonplaten @thomwolf 
transformers version == 4.0.0

I trained a T5 model for classification and I am trying to load the checkpoint model saved after 3 epochs. 
While trying to load the model, getting the below error.
![image](https://user-images.githubusercontent.com/54103522/102682082-a0281200-41ec-11eb-8292-4fbeba807ec3.png)


The model file is a ckpt file.

Any help is appreciated.
Thanks.


Comments:
Hi @adithyaan-creator 

Seems like you used `pytorch-lightning` for training, `pl` prefixes every `state_dict` key with `model.` so to load the `pl` checkpoint to hf format, remove the `model.` prefix from `state_dict` keys.

```python3
def remove_prefix(text: str, prefix: str):
    if text.startswith(prefix):
        return text[len(prefix) :]
    return text  # or whatever

state_dict = {remove_prefix(k, "model."): v for k, v in state_dict.items()}
hf_model.load_state_dict(state_dict)
hf_model.save_pretrained(save_path)
```
Also, would be nice if you could post code snippet instead of screenshots :)



Hi @patil-suraj,
I tried it with loading the 'state_dict' item to the model, and getting the following error "Unexpected key(s) in state_dict: "decoder.block.0.layer.1.EncDecAttention.relative_attention_bias.weight". ".

The code is :: 
def remove_prefix(text: str, prefix: str):
  if text.startswith(prefix):
        return text[len(prefix) :]
    return text  # or whatever

ckpt = {remove_prefix(k, "model."): v for k, v in ckpt['state_dict'].items()}

model.load_state_dict(ckpt)
model.save_pretrained("/content/model")`
  

Hey @adithyaan-creator,

You can ignore this warning, see: https://github.com/huggingface/transformers/pull/9231

@patrickvonplaten the code that was merged wasnt working out, but since the weight was unnecessary I deleted the weight layer and it worked out.
'del ckpt["decoder.block.0.layer.1.EncDecAttention.relative_attention_bias.weight"]'
Thanks.

@adithyaan-creator Hi, I am having the same issue now. Could you elaborate on how and where did you add 'del ckpt["decoder.block.0.layer.1.EncDecAttention.relative_attention_bias.weight"]' ? Thank you very much!
Error Loading a Hub Model (Multilingual-MiniLM)
Author: vyraun
Pull request: no

## Code Snippet

```
from transformers import AutoTokenizer, AutoModel
tokenizer = AutoTokenizer.from_pretrained("microsoft/Multilingual-MiniLM-L12-H384")
model = AutoModel.from_pretrained("microsoft/Multilingual-MiniLM-L12-H384")
```

- `transformers` version: 4.1.1, 3.1.0 (error in both)

## Error

```
TypeError: stat: path should be string, bytes, os.PathLike or integer, not NoneType
```

## Expected behavior

The [model and tokenizer](https://huggingface.co/microsoft/Multilingual-MiniLM-L12-H384) loads correctly. The error could be reproduced in a [colab notebook](https://colab.research.google.com/drive/1uFnBN-WdpK4PiamvdyizMCzJsrBtewKx?usp=sharing) .

Comments:
Please note: This checkpoint uses BertModel with XLMRobertaTokenizer so AutoTokenizer won't work with this checkpoint!
run_glue_no_trainer: datasets -> raw_datasets
Author: jethrokuan
Pull request: yes

# What does this PR do?

Use the correct variable (raw_datasets) instead of the module (datasets)
where appropriate. The script will otherwise fail with "module object is not subscriptable".

Fixes # (issue)

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?
@sgugger 

Comments:
Creating a quick PR now to alert to some issues I'm encountering, will backfill with proper GitHub issues after I knock off from work.

Other issues:
- [ ] If task_name is `None`, `metrics` is undefined, and will throw `... metrics used before assignment`
LayoutLM Tensorflow model
Author: atahmasb
Pull request: no

# 🚀 Feature request

It would be great if there was a TF version of the layoutlm. I see there are scripts in the repo to convert PyTorch checkpoints to TF models but I think the requirement is to have a TF model architecture to be able to load PyTorch model's wights in it.

## Motivation
We are using TF in production and we'd love to be able to use the layoutlm.

## Your contribution
I am happy to tackle the conversion. 

I was wondering if there are instructions on how to do the conversion properly so that it can be added to the repo. 


Comments:
Sure, I can guide you if you want.

As LayoutLM is only a slight adaptation from BERT, I guess you can define `modeling_tf_layoutlm.py` based on `modeling_tf_bert.py`.  Note that all layers should be renamed, e.g. `TFBertEmbeddings` -> `TFLayoutLMEmbeddings`. LayoutLM adds position embeddings for the tokens based on the bounding boxes, so I guess this is the only thing that needs to be added in the [embedding layer](https://github.com/huggingface/transformers/blob/88605f37a6fe7bde336f52700229d619b5ffa0f6/src/transformers/models/bert/modeling_tf_bert.py#L131). 

In PyTorch, we have:
https://github.com/huggingface/transformers/blob/88605f37a6fe7bde336f52700229d619b5ffa0f6/src/transformers/models/layoutlm/modeling_layoutlm.py#L65-L68

So this will need to be added to the `TFLayoutLMEmbeddings` class. Regarding a conversion script to convert the PyTorch weights into the TF version, there's a general script to convert PyTorch weights to TF 2 models: https://github.com/huggingface/transformers/blob/master/src/transformers/convert_pytorch_checkpoint_to_tf2.py

This works if all weights names are identical between the PT and TF implementations.

@NielsRogge Thanks for the guidance, I think I know where to start. 
I'll comment here if I had more questions. I am hoping to have a PR by the end of this week 

Should I upload TF weights under `https://huggingface.co/microsoft/` in the same place PT weights are stored?

Yes, pinging @julien-c here to give you access

pinging @LysandreJik and @sgugger 

We don't have granular write access to model repos so (unless you're affiliated with Microsoft in some way) I would suggest you upload the files to a model repo under your HF user account and then we (or them) can copy the relevant files to the main repos!

Let me know if this is a suitable workflow.

@julien-c Thanks, the suggested workflow sounds good.

closing this issue as the code is already merged.
Fix comment in modeling_t5.py
Author: lexhuismans
Pull request: yes

# What does this PR do?
This PR completes an incomplete comment in the modeling_t5.py file. 

`# ourselves in which case we just need to make it broadcastable to all heads.`
to 
```
# We can provide a self-attention mask of dimensions [batch_size, from_seq_length, to_seq_length] 
# ourselves in which case we just need to make it broadcastable to all heads.
```


Comments:
[doc] Custom datasets page reference dataset library as NLP library
Author: tomy0000000
Pull request: no

### Who can help

@sgugger 

## Information

The page [Fine-tuning with custom datasets](https://huggingface.co/transformers/custom_datasets.html) is referencing dataset library a lot, but in the old name (NLP library), and I've noticed that this still holds true in the [source .rst](https://github.com/huggingface/transformers/blob/master/docs/source/custom_datasets.rst) file.

If it wasn't left unmodified intended, I'm willing to help submitting a PR, just filling the issue to confirm.

Comments:
Yes, the name should indeed be updated! If you want to do a PR with this, please go ahead!
Add ImageFeatureExtractionMixin
Author: sgugger
Pull request: yes

# What does this PR do?

This PR adds a new `ImageFeatureExtractionMixin` to implement the common functionality needed for images (conversion to PIL Image / NumPy array /normalize/ resize) in a framework agnostic way. While it only adds support for torch (not tf) tensors as input, support for TF tensors is easy to add in the design and will be done when we have a TF model with a vision modality.

Along the way, this PR adds a new `is_vision_available` check (depends only on PIL for now, but we can add other dependencies later on if we fill we need them. It could for instance check for torchvision when torch is installed) and the "dummy" vision objects.

I will work on adding tests tomorrow, but the general design can already be reviewed to check if it has everything needed.
cc @NielsRogge 

Comments:
longformer speed compared to bert model
Author: gkim89
Pull request: no

We are trying to use a LongFormer and Bert model for multi-label classification of different documents.

When we use the BERT model (BertForSequenceClassification) with max length 512 (batch size 8) each epoch takes approximately 30 minutes.

When we use LongFormer (LongformerForSequenceClassification with the 'allenai/longformer-base-4096' and gradient_checkpointing=True) with max length 4096 (batch size 1, Gradient Accumulation step 8) each epoch takes approximately 12 hours.

Is this reasonable or are we missing something?
Is there anything that we can try to make the training faster?

Comments:
Hi,

Is it possible to ask questions related to training on the [forum](https://discuss.huggingface.co/) rather than here? For example, all questions related to training LongFormer can be found [here](https://discuss.huggingface.co/search?q=longformer).

The authors of Transformers like to keep Github issues for bugs/feature requests.

Thank you. 

sure. thank you for the quick response
How much vRAM should I have for fine tuning DeBERTa v2 xxlarge?
Author: ngoquanghuy99
Pull request: no

I'm fine tuning DeBERTa v2 xxlarge with 1.5B parameters on Nvidia Tesla T4 (16GB vRAM) and it returns "CUDA out of memory".
How much vRAM is enough? 
@LysandreJik 

Comments:
I don't know the answer, but hoping that it works after this PR got merged https://github.com/huggingface/transformers/pull/10753

Do you already use deepspeed ?

No, i did not
Error while predicting on single sentence for token classification task
Author: saurabhhssaurabh
Pull request: no

Hi

I have fine-tuned BERT for NER task. I am predicting on the fine-tuned model as:

`output = self.tokenizer(text)
trainer = Trainer(model=self.model, tokenizer=self.tokenizer)
trainer.predict(output)`

This code snippet is throwing the following error:

  File "run_ner_test_3.py", line 486, in <module>
    obj.predict(text="i require to send 9330793.33 by account")
  File "run_ner_test_3.py", line 430, in predict
    trainer.predict(output)
  File "/home/dev01/python_3/lib/python3.6/site-packages/transformers/trainer.py", line 1596, in predict
    test_dataloader, description="Prediction", ignore_keys=ignore_keys, metric_key_prefix=metric_key_prefix
  File "/home/dev01/python_3/lib/python3.6/site-packages/transformers/trainer.py", line 1658, in prediction_loop
    for step, inputs in enumerate(dataloader):
  File "/home/dev01/python_3/lib64/python3.6/site-packages/torch/utils/data/dataloader.py", line 435, in __next__
    data = self._next_data()
  File "/home/dev01/python_3/lib64/python3.6/site-packages/torch/utils/data/dataloader.py", line 475, in _next_data
    data = self._dataset_fetcher.fetch(index)  # may raise StopIteration
  File "/home/dev01/python_3/lib64/python3.6/site-packages/torch/utils/data/_utils/fetch.py", line 44, in fetch
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File "/home/dev01/python_3/lib64/python3.6/site-packages/torch/utils/data/_utils/fetch.py", line 44, in <listcomp>
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File "/home/dev01/python_3/lib/python3.6/site-packages/transformers/tokenization_utils_base.py", line 305, in __getitem__
    return self._encodings[item]
IndexError: list index out of range


Can you please suggest how to predict on a single sentence?

Comments:
@saurabhhssaurabh `trainer.predict()` expects an instance of `torch.utils.data.Dataset` to be passed and not a single sentence. I think it will be easier to use the trained model at self.model to predict rather than trying to use trainer object, as it does not have any single prediction method yet.

@frankhart2018 
Thank you for replying. I will implement code using prediction from self.model.