noamaanMulla-03/transformers-issues huggingface dataset:
Missing head_mask and decoder_head_mask arguments in encoder-decoder models
Author: stancld
Pull request: no

# 🚀 Feature request

Following the PRs #9569, #9634 and #9639, there are other encoder-decoder models, which either do not support `head_mask` and `decoder_head_mask` input arguments at all or can be only provided with a single `head_mask` argument used for head masking both in encoder and decoder. It would be, therefore, nice to make this feature uniform over all the decoder-models.

<hr>

**Models:**

| Model | Pytorch | TensorFlow | PR | Copy dependency |
| ------ | :------: | :---------: | :--: | :-----: |
| BERTGeneration | ☑️  | ✖️  | - | - |
| EncoderDecoderModel | ☑️  | ✖️  | - | - |
| FSMT | ✅ | ✖️  | #9819 | - |
| LED | ✅ | ☑️ | PT - #9856 ; TF - #9988 | - |
| ProphetNet | ☑️  | ✖️  | #9964 | - |
| Longformer | ✅  | ☑️  | PT - #9856; TF - #9988  | LED |


## Your contribution

I'm happy to add this feature in the following days, both for PyTorch and TensorFlow models. (Likely in shorter PRs in order not to create large, overwhelming PRs)

<hr>

Reviewers: @patrickvonplaten, @jplu, @sgugger, @LysandreJik, @stas00 .

Comments:
Does the latest huggingface-transformers version work with tokenizers==0.10.0?
Author: unclebob7
Pull request: no

## Environment info

- `transformers` version: 4.3.0.dev0
- `tokenizers` version: 0.10.0
![image](https://user-images.githubusercontent.com/29818505/104864385-c5d34300-5973-11eb-9e26-d804642e9331.png)


Comments:
Hello! We're [still pinned to 0.9.4](https://github.com/huggingface/transformers/blob/master/setup.py#L134). We'll pass to `0.10.0` soon.

@LysandreJik , if possible, could you provide your best guess as to when this requirement will be updated? I'm currently need to train a WordLevel Tokenizer, as well as use a transformer's model in the same process. I've broken the process up into two python files, which I run separately with different python environments, but it would be nice to have the full process in one file using one environment. Thanks!

@n1t0 tells me it should be ready sometimes next week!
Added an integration test for the Pytorch implementation of the DistilBERT model from issue #9948
Author: danielpatrickhug
Pull request: yes


# Adds Integration testing  for pytorch implementation of DistilBert from issue #9948

I implemented the test as described in the issue linked. I ran the test and it passed. I can extend the tests after confirmation of this current PR. Please let me know what you think. Thank you


Fixes #9948 


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [x] Did you write any new necessary tests?






Comments:
Hello! Why did you close your branch? The integration test looks good, you only need to run `make fixup` at the root of your clone to apply the quality requirements.

@LysandreJik Hi, sorry I was getting an error when i ran 'make fixup' and I was trying to figure it out. Ill finish it up tonight, unless you know whats wrong? Thank you for responding.

```
  File "utils/get_modified_files.py", line 28
    modified_files = subprocess.check_output(f"git diff --name-only {fork_point_sha}".split()).decode("utf-8").split()
                                                                                    ^
SyntaxError: invalid syntax
No library .py files were modified
  File "setup.py", line 192
    entries = "\n".join([f'    "{k}": "{v}",' for k, v in deps.items()])
                                            ^
SyntaxError: invalid syntax
```
Add `from_slow` in fast tokenizers build and fixes some bugs
Author: sgugger
Pull request: yes

# What does this PR do?

This PR adds an argument to the initialization of the `PreTrainedTokenizerFast` to force the conversion from a slow tokenizer. This will be useful to help users re-build the `tokenizer.json` file for some models where we can't update faulty ones right now without breaking backward compatibility (see #9637).

In passing it fixes a few bugs:
- wrong formatting for the documentation
- the fast sentencepiece tokenziers don't have an `sp_model` attribute so remove the documentation for that
- BarthezTokenizerFast was not registered properly in the autotokenizers, so `AutoTokenizer` was not finding it

Comments:
Add BartForCausalLM analogs to `ProphetNetForCausalLM`
Author: patrickvonplaten
Pull request: no
Labels: name: Good First Issue

name: Good Second Issue

# 🚀 Feature request

Bart is a seq2seq model, but there might be applications where one would like to use only the pre-trained BartDecoder in an EncoderDecoder setting with a "long" encoder, such as 

```python

from transformers import EncoderDecoderModel

model = EncoderDecoderModel("allenai/longformer-large-4096", "facebook/bart-large")

# fine-tune model ...
```

This is already possible for ProphetNet:

```python
from transformers import EncoderDecoderModel
import torch

model = EncoderDecoderModel.from_encoder_decoder_pretrained("allenai/longformer-large-4096", "microsoft/prophetnet-large-uncased")

input_ids = torch.tensor([10 * [1]])
labels = torch.tensor([10 * [0]])
loss = model(input_ids, decoder_input_ids=labels, labels=labels).loss
loss.backward()
```


, but not yet for Bart. This "Good first/second issue" is about implemented a `BartForCausalLM` analogs to the one in ProphetNet here:
https://github.com/huggingface/transformers/blob/9cc9f4122e2a1027a6011951e3c6629a0f1b6c3e/src/transformers/models/prophetnet/modeling_prophetnet.py#L1882

To verify that the feature works as expected, one should make sure that the following tests are added:
- A `BartStandaloneDecoderModelTest` class as is done in https://github.com/huggingface/transformers/blob/9cc9f4122e2a1027a6011951e3c6629a0f1b6c3e/tests/test_modeling_prophetnet.py#L1072
- And an encoder-decoder test class as it's done here: https://github.com/huggingface/transformers/blob/9cc9f4122e2a1027a6011951e3c6629a0f1b6c3e/tests/test_modeling_encoder_decoder.py#L758


<!-- A clear and concise description of the feature proposal.
     Please provide a link to the paper and code in case they exist. -->

## Motivation

mentioned above for Long-range seq2seq warm-starting e.g.

## Your contribution

I'm more than happy to guide someone through this issue!
It's a bit more advanced so I'll give it both "Good first issue" and "Good second issue".

You can claim the issue by writing it below and/or opening a PR :-) 


Comments:
I dont know if @MeRajat claimed this issue, However if not **I want to take this issue** 

Usually one opens a PR to claim the issue (the PR does not have to be finished) - so I think it's still open. 

@patrickvonplaten This FR is opened for some time,so thought of working on it, almost completed development, should I raise PR?
As @sadakmed is also working on it, so thought of asking.

Hey @spatil6,

I think the PR is already in an advanced stage, so I hope the PR is finished by next week. If not, I'll ping you again :-)
[Good first issue] DistilBERT PyTorch Integration tests
Author: LysandreJik
Pull request: no
Labels: name: PyTorch

name: Tests

name: Good First Issue

The PyTorch implementation of the DistilBERT model currently has no integration tests. This is problematic as the behavior can diverge without being noticed.

The [test_modeling_distilbert.py](https://github.com/huggingface/transformers/blob/master/tests/test_modeling_distilbert.py) file should be updated to include integration testing.

An example of a good modeling integration test is visible in the [test_modeling_bert.py#L552-L565](https://github.com/huggingface/transformers/blob/master/tests/test_modeling_bert.py#L552-L565) file:

https://github.com/huggingface/transformers/blob/1809de5165804666ba6c6a02a9d177f6683869cc/tests/test_modeling_bert.py#L552-L565

Some additional tips:
- The test must be marked as slow using the `@slow` decorator, so as to be run *daily*, and not on every commit of every branch/pull request of this repository.
- The test must be decorated with the `@require_torch` decorator so as to only be run in environments using PyTorch.
- A single test is necessary. If you feel like implementing multiple of these, then sharing the same checkpoint would be ideal so as to reduce download time.

Comments:
Adversarial/amnesic heads
Author: eritain
Pull request: no
Labels: name: Feature request

# 🚀 Feature request

Task heads that backpropagate deliberately reversed gradients to the encoder. A flag requesting this behavior when constructing a task head.

## Motivation

Transfer learning experiments lend themselves to questions about the extent to which two tasks rely on the same information about a word/sentence, and to experiments probing whether and how word encodings contain/correspond to syntax trees, lemmas, frequencies, and other objects of linguistic/psycholinguistic study.

A difficulty is that a pretrained model, without fine-tuning, may already encode certain information too thoroughly and accessibly for intermediate training to make much of a difference. For example, BERT's masked language modeling objective produces word encodings in which syntax information is readily accessible. Intermediate training on a syntax task requires training a task head to extract this information, of course, but it will result in very little reorganization of the encoder itself.

Adversarial training, such as the amnesic probing of Elazar et al. 2020, can avoid this pitfall. Intermediate training can aim to burn particular information *out* of the encodings, and measure how much this impairs trainability of the target task. Strictly reversing the sense of the training data won't do it though; getting all the answers exactly wrong requires just as much domain knowledge as getting them all right does. And randomizing the labels on training data may just result in a feckless task head, one that discards useful information passed to it from the encoder, rather than affecting the encoder itself. 

Ideally, then, the task head would be trained toward correctly reproducing gold-standard labels, but would flip all its gradients before backpropagating them to the shared encoder, thus training it not to produce precisely the signals that the task head found most informative. The following work by Cory Shain illustrates flipping gradients in this way (although it's not applied to shared-encoder transfer learning, but rather to development of encoders that disentangle semantics from syntax).

	https://docs.google.com/presentation/d/1E89yZ8jXXeSARDLmlksOCJo83QZdNbd7phBrR_dRogg/edit#slide=id.g79452223cd_0_19
	https://github.com/coryshain/synsemnet

## Your contribution

I am deeply unfamiliar with pytorch, unfortunately, and utterly ignorant of tensorflow. I can't offer much.

Comments:
Interesting thread, thank you for posting it! You could also post it on [the forums](https://discuss.huggingface.co) to reach more users!
Remove unintentional "double" assignment  in TF-BART like models
Author: stancld
Pull request: yes

This PR fixes unintentionally used "double" assignment during reshaping of `attn_wegihts` in the TF BART-like models.

**Description:** Replace `attn_weights = attn_wegihts = tf.reshape(...)` with `attn_weights = tf.reshape(...)` and thus remove unintentionally used "double" assignment.

<hr>

Reviewer: @jplu

Comments:
Added Integration testing for Pytorch implementation of DistilBert model from issue #9948'
Author: danielpatrickhug
Pull request: yes

# Adds Integration testing  for Pytorch implementation of DistilBert from issue #9948

*Redid pull request
*My environment wasn't set up right.



I implemented the test as described in the issue linked. I ran the test and it passed. I can extend the tests after confirmation of this current PR. Please let me know what you think. Thank you


Fixes #9948 


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [x] Did you write any new necessary tests?




Comments:
@LysandreJik Hey, Thank you for merging my first pull request. Happy to help! That's exactly what happened, took me a second to realize, but it became pretty clear when I read through the make file.
Authorize last version of tokenizer
Author: sgugger
Pull request: yes

# What does this PR do?

This PR bumps the version pinned in the setup to authorize the last version of tokenziers (which in particular contains fixes for the `run_qa` script).

Comments:
Yeah I forgot to add this in a comment -> Talked to @n1t0 about it and he says those are unwanted breaking changes, so we will pin to the next patch release he is going to make to tokenizers. Leaving the PR open in the meantime!

Added a few more things to this PR:
- Use last tokenizers RC release
- Update the conversion from slow to fast tokenizers as described in https://github.com/huggingface/transformers/issues/9637
- Added a script to verify the conversion from slow to fast tokenizers looks good
- Fix some links to the hub

Regarding the masks, should that be applied to all SentencePiece-based tokenizers? Should it be added to XLNet/ALBERT/T5 as well?

Can't approve since this is my PR originally, but this looks good to me. We just need to make sure all special masks tokens are taken into account.

Most of the tests are unrelated (a rebase on `master` will make them pass), but these two aren't:

```
FAILED tests/test_tokenization_pegasus.py::PegasusTokenizationTest::test_mask_tokens_rust_pegasus
FAILED tests/test_tokenization_mbart.py::MBartTokenizationTest::test_embeded_special_tokens
=== 2 failed, 3352 passed, 3280 skipped, 2512 warnings in 765.06s (0:12:45) ====
```
[trainer] a few fixes
Author: stas00
Pull request: yes

This PR:
- removes `model.to(device)` - it's not needed for DeepSpeed.  but primarily this allows loading models that otherwise won't load - e.g. loading 45GB (fp32) to a 40GB GPU when using Deepspeed with fp16 - as it loads only 22GB of it. But currently we load all 45GB right away and well nothing works
- decouples 2 unrelated logical things related to model parallel, which was very confusing in the previous if/else incarnation
- fixes a bug that left a deepspeed model to be  wrapped in DDP, but it shouldn't, like a few other bugs of the same kind I created as things just happened to work until they didn't. 

This PR enables t5-11b training on 1x 40GB gpu w/ Deepspeed  https://github.com/huggingface/transformers/issues/9996 

@sgugger

Comments:
This is breaking sadly: with this change someone using `trainer.model` after instantiating a `Trainer` won't have it on the GPU anymore, which will make code fail. It's also best IMO if an OOM error happens sooner rather than later.

Now for deepspeed I understand why this would be necessary, so we can move the `model.to` in that case. I don't see other cases when this is useful (mixed precision with APEX and AMP keep a copy of the model in full precision)

oh, that's no problem for now. Let's do it just for deepspeed then. Fairscale might join down the road.

Actually Deepspeed doesn't even need the `.to()` call at all. So it's even simpler.

So basically this skipping `.to()` is needed for all extensions that partition or tweak the model size, so MP/DeepSpeed and this will be so for PP as well.
ADD BORT
Author: stefan-it
Pull request: yes
Labels: name: PR for Model Addition

Hi,

this is a "clean" follow-up PR to the first attempt of adding Bort to Transformers (see #9112).

As Bort is based on the BERT architecture, there's no need to define own model classes, such as `BortModel`. This is done in the main Bort configuration via:

```json
"model_type": "bert"
```

Bort uses the same vocab as RoBERTa, so the tokenizer instance is also configured in the model configuration:

```json
"tokenizer_class": "RobertaTokenizer"
```

Basic integration tests and a (hopefully verbose) conversion script are also included in this PR.

Comments:
> Thanks for adding this new model! When referencing other pages in the documentation, it's better to use `:doc:` instead of a hard link, as it will then work in all versions of the documentation (which don't have the same base url).

Sorry that was my bad! I copied it from DialoGPT -> Updated it there as well

Ah sorry @patrickvonplaten, the `model_doc/` should be removed are the pages are in the same folder. That should resolve the build doc error.

Great job @stefan-it
fix steps_in_epoch variable in trainer when using max_steps
Author: yylun
Pull request: yes

# What does this PR do?

This PR fix the calculation of `steps_in_epoch` in `trainer.py` 

The 'step' in `steps_in_epoch` means one backward
The 'step' in `max_steps` means one parameter updating (taking gradient accumulation into account)

This bug does not affect training process, just make logging info weired

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Maybe @sgugger will be more interested in this

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- albert, bert, xlm: @LysandreJik
- blenderbot, bart, marian, pegasus, encoderdecoder,  t5: @patrickvonplaten, @patil-suraj
- longformer, reformer, transfoxl, xlnet: @patrickvonplaten
- fsmt: @stas00
- funnel: @sgugger
- gpt2: @patrickvonplaten, @LysandreJik
- rag: @patrickvonplaten, @lhoestq
- tensorflow: @jplu

Library:

- benchmarks: @patrickvonplaten
- deepspeed: @stas00
- ray/raytune: @richardliaw, @amogkam
- text generation: @patrickvonplaten
- tokenizers: @n1t0, @LysandreJik
- trainer: @sgugger
- pipelines: @LysandreJik

Documentation: @sgugger

HF projects:

- nlp datasets: [different repo](https://github.com/huggingface/nlp)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Examples:

- maintained examples (not research project or legacy): @sgugger, @patil-suraj
- research_projects/bert-loses-patience: @JetRunner
- research_projects/distillation: @VictorSanh

 -->


Comments:
And remove a repeated sentence in README
[Question] Pipeline QA start_index
Author: ahnz7
Pull request: no

I got a dictionary with 'score', 'start', 'end' and 'answer'. I want to use the 'start' and 'end' index. But the index of 'start' and 'end' are usually greater than length of encoding? Code:
text = text
question = quesiton
pipeline = pipeline('question-answering' , model = model)
answers = pipeline(context = data, question = questions)
print(answers)
{'score': 0.7909670472145081, 'start': 6192, 'end': 6195, 'answer': '111'}
but the length of context + question is smaller than 6000. How can I use 'start_index' and 'end_index' to verify that the result is '111'?

Comments:
Hi! So that we may help you, could  you provide the information related to your environment as asked in the issue template?

Also, we can't understand what's happening here because we don't know what's your text, your question, and most importantly the model you used.

Finally, you instantiated a model, but you didn't do so for the tokenizer. If you have mismatched models/tokenizers, then outputs are bound to be confusing.

> Hi! So that we may help you, could you provide the information related to your environment as asked in the issue template?
> 
> Also, we can't understand what's happening here because we don't know what's your text, your question, and most importantly the model you used.
> 
> Finally, you instantiated a model, but you didn't do so for the tokenizer. If you have mismatched models/tokenizers, then outputs are bound to be confusing.

Oh,sorry. My enviroonment is as follows:
OS: Macos catalina
python version: python3.7.3
Package Version: transformers 3.4 
The model, which I have used, is 'ktrapeznikov/biobert_v1.1_pubmed_squad_v2'. And the code is like following:

```py
data ='I live in Berkeley. I am 30 years old. And my name is Clara.'
question = 'What's my name?'
pipeline = pipeline('question-answering',model = 'ktrapeznikov/biobert_v1.1_pubmed_squad_v2')
answers = pipeline(context = data, question = question)

answer is like {'score': 0.9977871179580688, 'start': 54, 'end': 59, 'answer': 'Clara.'}


tokenizer = AutoTokenizer.from_pretrained('ktrapeznikov/biobert_v1.1_pubmed_squad_v2')
encoding = tokenizer.encode(questions,data)

len(encoding)
```

With len(encoding) I got the length of encoding 26. But from the answer I got start index as 54 and end index as 59?

Yes, the length of encoding is the length of the list of tokens. The start and end index are the start and end index of characters, not tokens. We should clarify that in the docs.

> Yes, the length of encoding is the length of the list of tokens. The start and end index are the start and end index of characters, not tokens. We should clarify that in the docs.

okay, thats what i mean. Thx