noamaanMulla-03/transformers-issues huggingface dataset:
Move the TF NER example
Author: jplu
Pull request: yes

# What does this PR do?

This PR moves the `run_tf_ner.py` example into the legacy folder because it uses the "legacy" way to train a model with the `utils_ner.py` file.


Comments:
Nice! What means "same way of training", same way than what?

Same way as the current `run_ner` script.
bug in bert pretraining 
Author: dorost1234
Pull request: no

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 4.3.2
- Platform: linux
- Python version: 3.7 
- PyTorch version (GPU?): 1.7 
- Tensorflow version (GPU?): - 
- Using GPU in script?: - 
- Using distributed or parallel set-up in script?: - 

### Who can help
<!-- Your issue will be replied to more quickly if you can figure out the right person to tag with @
 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- albert, bert, xlm: @LysandreJik
- blenderbot, bart, marian, pegasus, encoderdecoder,  t5: @patrickvonplaten, @patil-suraj
- longformer, reformer, transfoxl, xlnet: @patrickvonplaten
- fsmt: @stas00
- funnel: @sgugger
- gpt2: @patrickvonplaten, @LysandreJik
- rag: @patrickvonplaten, @lhoestq
- tensorflow: @jplu

Library:

- benchmarks: @patrickvonplaten
- deepspeed: @stas00
- ray/raytune: @richardliaw, @amogkam
- text generation: @patrickvonplaten
- tokenizers: @n1t0, @LysandreJik
- trainer: @sgugger
- pipelines: @LysandreJik

Documentation: @sgugger

HF projects:

- nlp datasets: [different repo](https://github.com/huggingface/nlp)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Examples:

- maintained examples (not research project or legacy): @sgugger, @patil-suraj
- research_projects/bert-loses-patience: @JetRunner
- research_projects/distillation: @VictorSanh

 -->

- albert, bert, xlm: @LysandreJik

## Information

In this line https://github.com/huggingface/transformers/blob/e73a3e1891775a915846cc0f24b7e9a26d6688fb/src/transformers/data/data_collator.py#L381 you need to change 0.5 to 0.1 to match the description written that only in 10% you would like to change  tokens with randomly selected tokens. 


## To reproduce
Nothing to reproduce.

<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

## Expected behavior

probs should match BERT paper. 

Comments:
Duplicate of https://github.com/huggingface/transformers/issues/10285
BertForSequenceClassification  -> TFBertForSequenceClassification causes 'bert.embeddings.position_ids' not used error
Author: dslim23
Pull request: no
Labels: name: wontfix

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->
     
- `transformers` version: 3.3.1
- Platform: Linux-4.9.0-13-amd64-x86_64-with-debian-9.13
- Python version: 3.7.8
- PyTorch version (GPU?): 1.4.0 (True)
- Tensorflow version (GPU?): 2.3.1 (True)
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: No

### Who can help
<!-- Your issue will be replied to more quickly if you can figure out the right person to tag with @
 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.
 
 albert, bert, GPT2, XLM: @LysandreJik 

 -->

## Information

Model I am using (Bert, XLNet ...):

```
from transformers import BertTokenizer, TFBertModel, TFBertForSequenceClassification,BertForSequenceClassification
import tensorflow as tf

tokenizer = BertTokenizer.from_pretrained('bert-base-cased')
model = TFBertForSequenceClassification.from_pretrained('output', from_pt=True)

```

I'm loading TFBertForSequenceClassification from a BertForSequenceClassification pytorch SavedModel and get this error: 

`Some weights of the PyTorch model were not used when initializing the TF 2.0 model TFBertForSequenceClassification: ['bert.embeddings.position_ids']`

I've done this before with this exact model and not had these issues. 


Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


Hi, I'm still getting this warning on version 4.3.2.
Clean TF ConvBert
Author: jplu
Pull request: yes

# What does this PR do?

This PR aims to clean TF ConvBert by adding explicit keyword arguments, typing and update the documentation in the model implementation to make it easier to understand and read.


Comments:
I don't understand this PR, I disagree with your proposal of adding keyword names when they're not required and don't help readability.

Mentioned here https://github.com/huggingface/transformers/pull/9788#discussion_r564365003 and here https://github.com/huggingface/transformers/pull/9788#discussion_r564373664.

Same goes for the BART refactor.

I have applied the same changes than in the #9788 PR. Should I remove all the keyword names and keep only the typing parts?
[examples/seq2seq] defensive programming + expand/correct README
Author: stas00
Pull request: yes

This PR deals with the new s2s script and its usage - mostly documentation.

This PR:

`run_seq2seq.py`:
* checks for invalid column names

`README.md`:
* largely expands the document explaining and exemplifying the supported formats
* documents the nuances of t5 and mbart translation - I hope we fix this on the programmatical level in the future 
* fixes examples where scores were bad - all examples were verified to work and provide good scores, including the custom files, which were far from easy to figure out. Hopefully now it'll be easier.
* makes the examples quick to complete by running only a short sample - this is important to notice breakages, e.g. in eval stage - nobody is going to wait for train to complete in hours.
* adds cnn/daily mail dataset
* recovers one preprocessed dataset from the last s2s incarnation recommendation: it is offered for high bleu scores (the other 3 are either identical or are just slightly worse than the preprocessed ones - full porting status: https://github.com/huggingface/transformers/issues/10044)

@patil-suraj, @sgugger 

Comments:
[Trainer] implement gradient_accumulation_steps support in DeepSpeed integration
Author: stas00
Pull request: yes
Labels: name: DeepSpeed

This PR:

Fixes in a bug:
- `lr_scheduler.step()` shouldn't be called under DeepSpeed - it's already called in its `optimizer.step()` internally - so it was moving through the scheduler rate change at twice the speed :(

Adds support for `gradient_accumulation_steps`:
* makes `gradient_accumulation_steps` work with deepspeed - for nuances see: https://github.com/microsoft/DeepSpeed/issues/776 - it required a lot of `if` / `if nots` - not helping the readability of the trainer  - and took a lot of trial and error to figure out - but what to do
* adds a corresponding doc 
* adds a first serious quality test for DeepSpeed that measures that `gradient_accumulation_steps` works - modelled after `test_trainer.py`'s own `test_gradient_accumulation` and extends it to compare loss as well, and also tests that the optimizer actually kicked in - with fp16 deepspeed it normally takes a few dozen steps before it kicks in with dynamic scaling enabled.
* extends `testing_utils` with a `mockenv_context` which is similar to `@mockenv`, but which can be used inside the test as context manager if multiple env vars need to be tested - `@mockenv` is only useful as a decorator. At the end I think I don't really need it as using the same env worked for all tests, but it might come handy if ports don't get released fast enough and then the test will use different ports - I'm concerned about CIs. And it's easier to re-use the class-wide env, rather hardcoding or creating a global variable - so it's just cleaner too.

Suggestion/Question:
* `get_regression_trainer` is very awesome! But importing it from a test file is not great - probably should move it and its components into a utilities file - `testing_utils.py` or create a new one `testing_training_utils.py` and in the future add other trainer-testing specific utils in there? Though this should be dealt with in a separate PR.

@sgugger 

Comments:
> Cool that they added it! This all looks pretty good to me!

Well, it has been there all this time, this PR just bolts it on correctly.

> Absolutely no problems with moving the regression trainer somewhere accessible, it was just in `test_trainer` because only used there.

Ah, that makes sense. I will rework it then next time I touch on this code.

Thank you for the feedback, @sgugger
[trainer] add Trainer methods for metrics logging and saving 
Author: stas00
Pull request: yes

This PR introduces:
* [x] `trainer.log_metrics` - to perform consistent formatting for logged metrics
* [x] `trainer.save_metrics` - to save the metrics

This removes a lot of pointless noise from the example scripts and makes them much easier to read and understand. It doesn't take away from a user understanding the example, since these helper methods are just removing formatting and file saving. 

If accepted it should be easy to replicate to other example scripts so that they all produce a consistent output and are all easier to read.

@sgugger 

Comments:
@sgugger, are you ok if we merge this and I will ask Second Good Issue to help with this - I'm not sure I will have time to do this and test all the scripts in the coming days, and since you guys discuss changing this script again, we should probably merge this first.

I'm fine with that!

Started an issue here: https://github.com/huggingface/transformers/issues/10337 - This is an easy task so I think First Good Issue might work. Let me know if I should bump it to Second.
[predict] AttributeError: 'Seq2SeqTrainer' object has no attribute 'metrics_format'
Author: Tan1997
Pull request: no

Hi everybody

When using mbart for machine translation prediction, i got:

Traceback (most recent call last):
  File "/Users/lishuqi/Desktop/WAT2021/transformers-master/examples/seq2seq/run_seq2seq.py", line 667, in <module>
    main()
  File "/Users/lishuqi/Desktop/WAT2021/transformers-master/examples/seq2seq/run_seq2seq.py", line 637, in main
    metrics_formatted = trainer.metrics_format(metrics)
AttributeError: 'Seq2SeqTrainer' object has no attribute 'metrics_format'

Am I doing something wrong with the translation?
@patil-suraj 



Comments:
`metrics_format` was recently introduced on master, you should update the transformers version to master.

Thanks! I'll try it again!
tokenizer.Tokenizer compatibility with Inference API or Auto* classes
Author: gstranger
Pull request: no
Labels: name: Feature request

# 🚀 Feature request
Make tokenizers created with the tokenizers library compatible with the Inference API or Auto classes

## Motivation

I have trained a model on a specific domain by modeling a sequence generation problem as a language modeling problem to predict the next token in the set. The tokenizer associated with the model I used (TransformerXL) was not compatible with my domain since my tokens contained whitespace so I created my own using the `WordLevelTrainer` class in the `tokenizers` library. **Now that I have a complete working solution I would like to use this tokenizer and model in the huggingface Inference API, however it does not work because it requires the tokenizer associated with the model**. Making the `transformers` models compatible with `tokenizers` library could make all kinds of use cases outside of NLP possible with these libraries. 

## Your contribution

Is it possible to hack the saved config for a tokenizer created through the `tokenizers` library to work directly with the `Auto` classes? If so I can document this approach for other users.


Comments:
Adding RAG to text-generation pipeline
Author: lalitpagaria
Pull request: no
Labels: name: wontfix

# 🚀 Feature request

Thank you for the awesome work. I am working on https://github.com/deepset-ai/haystack/issues/443 and just wanted to check whether any plan to add RAG into `text-generation` pipeline.

## Motivation

`text-generation` already have other models, hence it I would be great to have it in there. And this will help keeping our code clean by not adding classes for each type of generators.

```
model = pipeline('text-generation', model="facebook/rag-token-nq", tokenizer=None, device=-1)
# ValueError: Unrecognized configuration class <class 'transformers.configuration_rag.RagConfig'> for this kind of AutoModel: AutoModelForCausalLM.
Model type should be one of CamembertConfig, XLMRobertaConfig, RobertaConfig, BertConfig, OpenAIGPTConfig, GPT2Config, TransfoXLConfig, XLNetConfig, XLMConfig, CTRLConfig, ReformerConfig, BertGenerationConfig.
```

## Your contribution

If you guide me I am happy to help.


Comments:
Hey @lalitpagaria - RAG is quite different to other generation models so we don't have it on the short-term roadmap to add it to pipelines. We are still thinking about how to integrate retrieval augmented models to the pipelines.

Thanks @patrickvonplaten 
Yeah I totally agree with you.
Please let me know whether I close this issue or keep it open for future reference.

Leave it open - I'll put it under projects so that I don't forget it :-) 


This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


@patrickvonplaten Great work! I noticed that `transformers` included the implementation for `DPR`. But for `RAG`, I only find a [demo](https://huggingface.co/rag/). Is there a source code for `RAG`? Or do you know where is Facebook's source code for `RAG`? 

transformers does include RAG.
You can even find the documentation here: https://huggingface.co/transformers/model_doc/rag.html
Can every line in the input CSV file contain more than one sentence when pertraining BERT for MLM Loss?
Author: abhisheksgumadi
Pull request: no

Hello HF Team,

  I am familiar with how to pretrain BERT and I have a Dataloader that reads an input CSV file line by line and every time it reads a line, it tokenizes it and sends back the tokens for training to the training code. My question is whether is it ok for this input CSV file to contain more than one sentence on every line when pretraining BERT for masked language modelling? 
  
Otherwise, is it important for it to contain only one meaningful sentence only? I am thinking if self attention will still continue to work and the model train properly even if every single line in the input CSV file (single training sample) is actually more than one sentence, each of it separated with a '.' delimiter of course. 

Thanks

Comments:
Hello, thanks for opening an issue! We try to keep the github issues for bugs/feature requests.
Could you ask your question on the [forum](https://discusss.huggingface.co) instead? You'll get more answers over there, as questions like these are the point of the forum :)

Thanks!
Where can we find the `RAG` implementation?
Author: g-jing
Pull request: no

I noticed that `transformers` included the implementation for `DPR`. But for `RAG`, I only find a [demo](https://huggingface.co/rag/). Is there a source code for `RAG`? Or do you know where is Facebook's source code for `RAG`? Thanks

Comments:
The implementation can be found [here](https://github.com/huggingface/transformers/tree/master/src/transformers/models/rag).

@NielsRogge Thanks! That's it.
[Benchmark]
Author: IsaacZachary
Pull request: no

# 🖥 Benchmarking `transformers`

## Benchmark

Which part of `transformers` did you benchmark?

## Set-up

What did you run your benchmarks on? Please include details, such as: CPU, GPU? If using multiple GPUs, which parallelization did you use?

## Results

Put your results here!

Comments:
BERT LOSS FUNCTION
Author: alshahrani2030
Pull request: no
Labels: name: wontfix

My question is that can I use KLDivLoss instead of CrossEntropyLoss when I fine-tune BERT for classification? the reason for that is that I want to pass the weight of each class(e.g for binary classification, instead of 1 or 0 I will pass the probability distribution )



Thank you in advance

Comments:
Sure you can do that. Create a class which inherits from [BertForSequenceClassification](https://github.com/huggingface/transformers/blob/master/src/transformers/modeling_bert.py#L1122) and overwrite the [forward](https://github.com/huggingface/transformers/blob/master/src/transformers/modeling_bert.py#L1134) method.

Instead of overwriting the forward method you can retrieve the hidden states and compute the loss as you would do with any PyTorch model. 

The loss is only computed by the model when you hand the `labels` to the model, which is not a required argument.

> Instead of overwriting the forward method you can retrieve the hidden states and compute the loss as you would do with any PyTorch model.
> 
> The loss is only computed by the model when you hand the `labels` to the model, which is not a required argument.

Could you please elaborate on the same please? @LysandreJik 


What do you want me to elaborate on?

> Instead of overwriting the forward method you can retrieve the hidden states and compute the loss as you would do with any PyTorch model.
This @LysandreJik 

I got this error when I called 
loss.backward()
 loss is "torch.float64" type


![error-KLDiv](https://user-images.githubusercontent.com/55197626/81520607-c7436a80-9312-11ea-91ae-85fb7f032d96.PNG)


This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


> My question is that can I use KLDivLoss instead of CrossEntropyLoss when I fine-tune BERT for classification? the reason for that is that I want to pass the weight of each class(e.g for binary classification, instead of 1 or 0 I will pass the probability distribution )
> 
> Thank you in advance

Hi did you manage to do this? I also need to pass class probability distribution instead of the labels and am not sure how to do this.

BertForSequenceClassification.forward() returns the logits also. You can use these in any pytorch loss function (eg: KLDivLoss, not sure if you'll need to softmax them first) and then run backward on the resulting loss. It's a bit redundant (since BertForSequenceClassification's loss is still calculated), but works.

> Sure you can do that. Create a class which inherits from [BertForSequenceClassification](https://github.com/huggingface/transformers/blob/master/src/transformers/modeling_bert.py#L1122) and overwrite the [forward](https://github.com/huggingface/transformers/blob/master/src/transformers/modeling_bert.py#L1134) method.

That link to the forward function is stale, as of Feb 24 it's [here](https://github.com/huggingface/transformers/blob/7e662e6a3be0ece455b4c4ae2c3348beab11bad5/src/transformers/models/bert/modeling_bert.py#L1475).
DialoGPT tokenizer config issue
Author: ayubSubhaniya
Pull request: no

## Environment info

- `transformers` version: 4.3.2
- Platform: Darwin-19.3.0-x86_64-i386-64bit
- Python version: 3.6.5
- PyTorch version (GPU?): 1.6.0 (False)
- Tensorflow version (GPU?): not installed (NA)
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: No

### Who can help
Models:

- gpt2: @patrickvonplaten, @LysandreJik

Library:
- tokenizers: @n1t0, @LysandreJik

## Information

Model I am using (DialoGPT-small.):
When I load tokenizer, its `model_max_len` is coming infinite.

```
from transformers import AutoModelForCausalLM, AutoTokenizer
import torch


tokenizer = AutoTokenizer.from_pretrained("microsoft/DialoGPT-small")
model = AutoModelForCausalLM.from_pretrained("microsoft/DialoGPT-small")

max_len = tokenizer.model_max_length
```

## Expected behavior
Before It was coming as 1024. 

Is this some recent change?


Comments:
Hi! This change may have originated from the move to the git-based repos. @patrickvonplaten and I have just modified the DialoGPT tokenizer configuration to have 1024 as model max length, you shouldn't have to do anything but re-run your script to see it updated.

@LysandreJik Don't hesitate to reference the url for the commit on huggingface.co for future reference

Here I believe it's https://huggingface.co/microsoft/DialoGPT-small/commit/9fb5c2d6a01395898bfd90acce2dbec1537730f1

Indeed! Here are the commits:

`DialoGPT-small`: [huggingface@364722e](https://huggingface.co/microsoft/DialoGPT-small/commit/364722ef15f5c04dcb9a57d3b77815bbc1d51efc) and [huggingface@9fb5c2d](https://huggingface.co/microsoft/DialoGPT-small/commit/364722ef15f5c04dcb9a57d3b77815bbc1d51efc)
`DialoGPT-medium`: [huggingface@e84a3e](https://huggingface.co/microsoft/DialoGPT-medium/commit/e84a3e0adc90aabc6b57e59318e15bf4b733eedc)
`DialoGPT-large`: [huggingface@acc7ea](https://huggingface.co/microsoft/DialoGPT-large/commit/acc7eaf98122bc6922976182b4d365d650f179b3)

Thanks @LysandreJik for quick fix.
Knowledge Retrieval missing from BlenderBot Implementation
Author: applyinnovations
Pull request: no
Labels: name: New model

# 🚀 Feature request

<!-- A clear and concise description of the feature proposal.
     Please provide a link to the paper and code in case they exist. -->

The original Blenderbot [paper](https://arxiv.org/pdf/2004.13637.pdf) considered three transformer based models (Retrieval, Generator and RetNRef), however from what I can see only the generator model is implemented within this repository: [transformers/src/transformers/models/blenderbot/modeling_blenderbot.py](https://github.com/huggingface/transformers/blame/master/src/transformers/models/blenderbot/modeling_blenderbot.py).

## Motivation

<!-- Please outline the motivation for the proposal. Is your feature request
     related to a problem? e.g., I'm always frustrated when [...]. If this is related
     to another GitHub issue, please link here too. -->

As part of my academic work I am generating topic bound conversations and wish to compare Blenderbot as well as make modifications to its knowledge retrieval component. It would be useful if someone could point me towards this (if it is already implemented) or inform me if this feature is planned to be added in the future. 

![image](https://user-images.githubusercontent.com/77219112/108953393-d8e5db00-76b6-11eb-8b26-4ee0b123755d.png)

## Contribution

<!-- Is there any way that you could help, e.g. by submitting a PR?
     Make sure to read the CONTRIBUTING.MD readme:
     https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md -->

Prior to making a contribution I want to confirm that feature is not implemented and there is no intention of implementing this in the near future.

Thanks,
Alex

@patrickvonplaten @patil-suraj 

Comments: