noamaanMulla-03/transformers-issues huggingface dataset:
Add Linformer model
Author: AaronFriel
Pull request: no
Labels: name: wontfix

name: New model

# 🌟 New model addition

## Model description

### Linformer: Self-Attention with Linear Complexity

Paper published June 9th on ArXiv: https://arxiv.org/abs/2006.04768

Large transformer models have shown extraordinary success in achieving state-of-the-art results in many natural language processing applications. However, training and deploying these models can be prohibitively costly for long sequences, as the standard self-attention mechanism of the Transformer uses O(n²) time and space with respect to sequence length. In this paper, we demonstrate that the self-attention mechanism can be approximated by a low-rank matrix. We further exploit this finding to propose a new self-attention mechanism, which reduces the overall self-attention complexity from O(n²) to O(n) in both time and space. The resulting linear transformer, the **Linformer**, performs on par with standard Transformer models, while being much more memory- and time-efficient. 

## Open source status

* [ ] the model implementation is available: (give details)
* [ ] the model weights are available: (give details)
* [x] who are the authors: Sinong Wang, Belinda Z. Li, Madian Khabsa, Han Fang, Hao Ma

Comments:
Here is an pytorch implementation
https://github.com/tatp22/linformer-pytorch

Just another implementation by the authors
https://github.com/facebookresearch/pytext/pull/1407


This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


Any Tensorflow implementation?
Command "python setup.py egg_info" failed with error code 1 in /tmp/pip-install-Q_fyRn/sacrebleu/
Author: rabeehk
Pull request: no

Hi
I am trying with master branch getting this error when installign requirements inside examples 
thanks. 

Collecting seqeval (from -r ../requirements.txt (line 3))
  Downloading https://files.pythonhosted.org/packages/9d/2d/233c79d5b4e5ab1dbf111242299153f3caddddbb691219f363ad55ce783d/seqeval-1.2.2.tar.gz (43kB)
    100% |████████████████████████████████| 51kB 13.4MB/s 
Collecting psutil (from -r ../requirements.txt (line 4))
  Downloading https://files.pythonhosted.org/packages/33/e0/82d459af36bda999f82c7ea86c67610591cf5556168f48fd6509e5fa154d/psutil-5.7.3.tar.gz (465kB)
    100% |████████████████████████████████| 471kB 2.7MB/s 
Collecting sacrebleu (from -r ../requirements.txt (line 5))
  Downloading https://files.pythonhosted.org/packages/b9/d6/258a1e63463b4731a387f0872dca759c330bf4845cc0464f2c65028674b6/sacrebleu-1.3.7.tar.gz
    Complete output from command python setup.py egg_info:
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/tmp/pip-install-Q_fyRn/sacrebleu/setup.py", line 65, in <module>
        version = get_version(),
      File "/tmp/pip-install-Q_fyRn/sacrebleu/setup.py", line 56, in get_version
        with open(os.path.join(os.path.dirname(__file__), 'sacrebleu.py'), encoding='utf-8') as fin:
    TypeError: 'encoding' is an invalid keyword argument for this function
    
    ----------------------------------------
Command "python setup.py egg_info" failed with error code 1 in /tmp/pip-install-Q_fyRn/sacrebleu/


Comments:
Installing transformers is also broken

(test) rabeeh@gpu4:~/transformers/examples/seq2seq$ pip install git+https://github.com/huggingface/transformers.git
Collecting git+https://github.com/huggingface/transformers.git
  Cloning https://github.com/huggingface/transformers.git to /tmp/pip-req-build-V7nNeF
  Installing build dependencies ... done
    Complete output from command python setup.py egg_info:
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/tmp/pip-req-build-V7nNeF/setup.py", line 156
        entries = "\n".join([f'    "{k}": "{v}",' for k, v in deps.items()])
                                                ^
    SyntaxError: invalid syntax
    
    ----------------------------------------
Command "python setup.py egg_info" failed with error code 1 in /tmp/pip-req-build-V7nNeF/


issue solved with python = 3.7
Extracting important information
Author: ghost
Pull request: no

I'm trying to extract important information from a lecture transcript. What's the best way to go about doing this ? This would be without a particular query parameter, just generally important information in the global context of the lecture.

Comments:
Hello, thanks for opening an issue! We try to keep the github issues for bugs/feature requests.
Could you ask your question on the [forum](https://discusss.huggingface.co) instead?

Thanks!
Transfoxl seq classification
Author: spatil6
Pull request: yes

This PR implements Sequence classification for Transformer XL model
TransfoxlForSequenceClassification uses the last token in order to do the classification, as other causal models (e.g. GPT-1,GPT-2) do.

Fixes #7623 (Partially)

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [x] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [x] Did you write any new necessary tests?


## Who can review?

@LysandreJik

Comments:
Where is "run_bert_classifier.py"?
Author: amirj
Pull request: no

Thanks for this great repo.
Is there any equivalent to [the previous run_bert_classifier.py](https://github.com/huggingface/pytorch-pretrained-BERT/tree/master/examples/run_bert_classifier.py)?


Comments:
It's now `run_glue.py`

Hi @thomwolf 
Doc needs changes from run_bert_classifier to run_glue
https://huggingface.co/pytorch-transformers/examples.html

Hey, just another headsup @thomwolf 
This Doc also needs changing for the run_bert_classifier.py:
https://huggingface.co/transformers/v1.1.0/examples.html#introduction

That's an old version of the doc @mtwright (notice the `v1.1.0`), you should check out the up to date one:

https://huggingface.co/transformers/examples.html#introduction

documentation is still broken, by the way

like, here, points to a bunch of files that do not exist. also not quite sure if these instructions work anymore:

https://huggingface.co/transformers/converting_tensorflow_models.html
Removing Head Layer/Model Conversion
Author: pugantsov
Pull request: no

I am currently working on some research in which I am to delve into the analysis of decision boundaries in text classification tasks and I am aiming to use recent work from the `ExBERT` paper, allowing me to visualise the importance of particular features across sentences.

Since the library is built on top of models from the Transformers library and requires that *The model architecture must be supported by the `AutoModelWithLMHead`*, I was wondering if it was possible to modify a fine-tuned model to work with that architecture. I am currently using `DistilBERTForSequenceClassification` in my pipeline and was wondering if it were possible to essentially fine-tune for a classification task and use the underlying `DistilBERT` model, as I assume all of the attention weights etc. will still be included in the model? ie. Could I change the loaded model after training to work with the library and to work with the `AutoModelWithLMHead` architecture so that I could inspect the attention heads?

I wasn't sure if I was only able to use models trained for Masked LM or if I could use models trained for downstream tasks? Apologies if this is a question best for the ExBERT github but since it was built into the library, thought I'd ask.

Comments:
A model trained for sequence classification can definitely be loaded with a different head. Here's an example:

```py
from transformers import DistilBertForSequenceClassification, DistilBertForMaskedLM
sequence_classifier = DistilBertForSequenceClassification.from_pretrained("...")
# Do stuff with your model, train it, do what you like

# Save the weights in a local directory
sequence_classifier.save_pretrained("model-trained-on-xxx")

# Load the weights in the *ForMaskedLM model.
language_model = DistilBertForMaskedLM.from_pretrained("model-trained-on-xxx")
```
This `language_model` has kept all the weights of the base transformer model, has discarded the sequence classification layers, and has randomly initialized the new layers. This model can be loaded in an `AutoModelWithLMHead`.

Oh wow, did not expect it to be this easy. Thanks very much!
Fix typo in docstring in src/transformers/models/bert_japanese/tokenization_bert_japanese.py
Author: ryota-mo
Pull request: yes

# What does this PR do?
Only fix typo (thi -> this).

## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?



@sgugger, @LysandreJik

Comments:
Corrected a typo in the ReadMe
Author: devangi2000
Pull request: yes

# What does this PR do?

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

Fixes # (issue)


## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [x] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors which may be interested in your PR.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

 albert, bert, XLM: @LysandreJik
 GPT2: @LysandreJik, @patrickvonplaten
 tokenizers: @mfuntowicz
 Trainer: @sgugger
 Benchmarks: @patrickvonplaten
 Model Cards: @julien-c
 examples/distillation: @VictorSanh
 nlp datasets: [different repo](https://github.com/huggingface/nlp)
 rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)
 Text Generation: @patrickvonplaten, @TevenLeScao
 Blenderbot, Bart, Marian, Pegasus: @patrickvonplaten
 T5: @patrickvonplaten
 Rag: @patrickvonplaten, @lhoestq
 EncoderDecoder: @patrickvonplaten
 Longformer, Reformer: @patrickvonplaten
 TransfoXL, XLNet: @TevenLeScao, @patrickvonplaten
 examples/seq2seq: @patil-suraj
 examples/bert-loses-patience: @JetRunner
 tensorflow: @jplu
 examples/token-classification: @stefan-it
 documentation: @sgugger
 FSMT: @stas00
 -->


Comments:
Thank you very much for this correction, @devangi2000
KeyError: 'mt5'
Author: astakara48
Pull request: no

I am trying to use the google/mt5 model, but I get KeyError: 'mt5'. How do I fix this?

KeyError                                  Traceback (most recent call last)
<ipython-input-2-207aa15555f1> in <module>
----> 1 tokenizer = AutoTokenizer.from_pretrained("google/mt5-small")

/usr/local/lib/python3.8/dist-packages/transformers/tokenization_auto.py in from_pretrained(cls, pretrained_model_name_or_path, *inputs, **kwargs)
    304         config = kwargs.pop("config", None)
    305         if not isinstance(config, PretrainedConfig):
--> 306             config = AutoConfig.from_pretrained(pretrained_model_name_or_path, **kwargs)
    307 
    308         if "bert-base-japanese" in str(pretrained_model_name_or_path):

/usr/local/lib/python3.8/dist-packages/transformers/configuration_auto.py in from_pretrained(cls, pretrained_model_name_or_path, **kwargs)
    334 
    335         if "model_type" in config_dict:
--> 336             config_class = CONFIG_MAPPING[config_dict["model_type"]]
    337             return config_class.from_dict(config_dict, **kwargs)
    338         else:

KeyError: 'mt5'

Comments:
pip install transformers==4.0.0rc1 sentencepiece


> pip install transformers==4.0.0rc1 sentencepiece

Thank you! You are my hero
Don't warn that models aren't available if Flax is available.
Author: skye
Pull request: yes

# What does this PR do?

Disables the "Neither PyTorch nor TensorFlow >= 2.0 have been found" warning if Flax has been found.

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors which may be interested in your PR.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

 albert, bert, XLM: @LysandreJik
 GPT2: @LysandreJik, @patrickvonplaten
 tokenizers: @mfuntowicz
 Trainer: @sgugger
 Benchmarks: @patrickvonplaten
 Model Cards: @julien-c
 examples/distillation: @VictorSanh
 nlp datasets: [different repo](https://github.com/huggingface/nlp)
 rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)
 Text Generation: @patrickvonplaten, @TevenLeScao
 Blenderbot, Bart, Marian, Pegasus: @patrickvonplaten
 T5: @patrickvonplaten
 Rag: @patrickvonplaten, @lhoestq
 EncoderDecoder: @patrickvonplaten
 Longformer, Reformer: @patrickvonplaten
 TransfoXL, XLNet: @TevenLeScao, @patrickvonplaten
 examples/seq2seq: @patil-suraj
 examples/bert-loses-patience: @JetRunner
 tensorflow: @jplu
 examples/token-classification: @stefan-it
 documentation: @sgugger
 FSMT: @stas00
 -->


Comments:
Avoid erasing the attention mask when double padding
Author: sgugger
Pull request: yes

# What does this PR do?

There is currently a bug when padding the same inputs twice:
```
>>> from transformers import AutoTokenizer
>>> tokenizer = AutoTokenizer.from_pretrained("bert-base-uncased")
>>> inputs = tokenizer("Hello, my name is Sylvain!", padding="max_length", max_length=32)
>>> print(inputs["attention_mask"])
[1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]
>>> inputs = tokenizer.pad(inputs, padding="max_length", max_length=32)
>>> print(inputs["attention_mask"])
[1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]
```

This is done when using the `DataCollatorWithPadding` inside a `Trainer` (which is the default) when the samples have already been padded. This PR fixes that by honoring the current `attention_mask` when no padding is necessary.

Comments:
Fine-tune with custom data
Author: whoafridi
Pull request: no

1. What is the difference between `run_squad.py` & `run_squad_trainer.py` ?

I've squad like dataset. 2. What script I used for fine-tuning with my own dataset? 

Comments:
`run_squad.py` is more complete right now, as `run_squad_trainer.py` can't do evaluation (yet! it will be possible in a few days).

We try to keep the github issues for bugs/feature requests.
For next time, could you ask your question on the [forum](https://discusss.huggingface.co) instead?

Thanks!

Okay sure. I don't know about the forum - sorry for that. Thank you @LysandreJik
Tweak wording + Add badge w/ number of models on the hub
Author: julien-c
Pull request: yes

it seemed pertinent to display this here but maybe we can also add it to some other places

Comments:
[Tooling comment] For some reason `python utils/check_copies.py --fix_and_overwrite` fails on my Python with following:
```
(.env) ipro:transformers gibbon$ python utils/check_copies.py --fix_and_overwrite
Traceback (most recent call last):
  File "utils/check_copies.py", line 432, in <module>
    check_model_table(args.fix_and_overwrite)
  File "utils/check_copies.py", line 414, in check_model_table
    new_table = get_model_table_from_auto_modules()
  File "utils/check_copies.py", line 328, in get_model_table_from_auto_modules
    spec = importlib.util.spec_from_file_location(
AttributeError: module 'importlib' has no attribute 'util'

```

I think it's better at the top rather than the end of the list (I don't think a user will read to until the end of the list TBH). We could even put it further up the README!
Impossible to use  sentencepiece
Author: lematmat
Pull request: no

Hi,

I explicitly installed both the latest version of transformers (v4.0.0) and Sentencepiece (v0.1.84) as it is specified as it is specified in the release history:
![Capture d’écran 2020-12-03 à 16 12 29](https://user-images.githubusercontent.com/19993147/101047903-7fbc4e80-3582-11eb-9e72-ada4a1b55366.png)



And the I I try to import MarianMT Tokeninzer  I have the following issue:

![Capture d’écran 2020-12-03 à 16 16 41](https://user-images.githubusercontent.com/19993147/101048715-26a0ea80-3583-11eb-9448-568a9e1ebe60.png)

So, any idea why  I 'm getting that issue ?

Best Regards,

Leman




Comments:
Have you tried restarting the kernel after installing `sentencepiece`?

Yes, I did, with:
!pip install --upgrade sentencepiece

Is it possible for you to share your notebook so that I may take a look?

I've just relaunched my notebook now, I don't have any issue now.

Thank you for your help
Regards

Glad it works for you now!
BertModel outputs string instead of tensor
Author: miguelwon
Pull request: no

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->
     
- `transformers` version: 4.0.0
- Platform: Linux-4.15.0-46-generic-x86_64-with-Ubuntu-16.04-xenial
- Python version: 3.6.7
- PyTorch version (GPU?): 1.7.0 (True)
- Tensorflow version (GPU?): not installed (NA)
- Using GPU in script?: <fill in>
- Using distributed or parallel set-up in script?: <fill in>

### Who can help
Sorry, no idea. 

## Information

Model I am using (Bert, XLNet ...): BERT

The problem arises when using:
* [x] my own modified scripts: (give details below)

The tasks I am working on is:
* [x] my own task or dataset: (give details below)

## To reproduce

```
import transformers
from transformers import BertModel, BertTokenizer

PRE_TRAINED_MODEL_NAME = 'bert-base-cased'
PATH_OF_CACHE = "some_path"

tokenizer = BertTokenizer.from_pretrained(PRE_TRAINED_MODEL_NAME,cache_dir = PATH_OF_CACHE)

sample_txt = 'When was I last outside? I am stuck at home for 2 weeks.'

encoding_sample = tokenizer.encode_plus(
  sample_txt,
  max_length=32,
  add_special_tokens=True, # Add '[CLS]' and '[SEP]'
  return_token_type_ids=False,
  padding=True,
  truncation = True,
  return_attention_mask=True,
  return_tensors='pt',  # Return PyTorch tensors
)

bert_model = BertModel.from_pretrained(PRE_TRAINED_MODEL_NAME,cache_dir = PATH_OF_CACHE)

last_hidden_state, pooled_output = bert_model(
  encoding_sample['input_ids'],
  encoding_sample['attention_mask']
)

print([last_hidden_state,pooled_output])
```

I'm getting this very odd behaviour where the output are two strings named from the variables:

```
(env) mwon@sebruno2:~/data-mwon/paperChega/src_classificador$ python test.py 
['last_hidden_state', 'pooler_output']
```


## Expected behavior

I expected to output a tensor with the hidden state of the last layer.


Comments:
Hi! Indeed, model outputs cannot be unpacked this way. It is mentioned in the [documentation](https://huggingface.co/transformers/main_classes/output.html#transformers.file_utils.ModelOutput). You can retrieve the items by unpacking them like this if you use the `.to_tuple()` method.

Oh, ok. Thanks and sorry for missing that. 

No problem, happy to help!