noamaanMulla-03/transformers-issues huggingface dataset:
cannot load "layoutlm-base-uncased"
Author: rzhao6
Pull request: no

hi, 
im trying to do "LayoutLMTokenizer.from_pretrained('layoutlm-base-uncased')", and got an error saying "OSError: Model name 'layoutlm-base-uncased' was not found in tokenizers model name list (microsoft/layoutlm-base-uncased, microsoft/layoutlm-large-uncased). We assumed 'layoutlm-base-uncased' was a path, a model identifier, or url to a directory containing vocabulary files named ['vocab.txt'] but couldn't find such vocabulary files at this path or url." 
thank you!

- `transformers` version: 3.3.1
- Python version: 3.7.7
- PyTorch version (GPU?): 1.6.0, GPU

@mfuntowicz


Comments:
Hi,

Yes the error message is right, you have to add "microsoft/" before the name:
```
LayoutLMTokenizer.from_pretrained('microsoft/layoutlm-base-uncased')
```

ah i see, i directly used the line in the example here https://huggingface.co/transformers/model_doc/layoutlm.html#layoutlmmodel

thank you for helping!
The input training data files (multiple files in glob format).
Author: kfkelvinng
Pull request: yes

Very often Corpus comes in [split files (book-large-p1.txt, book-large-p2.txt)](https://huggingface.co/datasets/bookcorpus); Also splitting large files to smaller files can prevent [language_modeling's tokenizer going out of memory](https://github.com/huggingface/transformers/blob/6303b5a7185fba43830db0cbb06c61861f57ddff/src/transformers/data/datasets/language_modeling.py#L67) in environment like Colab that does not have swap memory and limited to Standard (12Gb) or High RAM (25Gb) instances.

To avoid making any assumption and prematurely truncate the file to avoid such error, we add support to concatenate training data on Dataset level. User can split files to multiple 512Mb, in that case language_modeling's tokenizer would be less to go out of memory.

In addition, it would even further enhance the memory limitation if we keep [pytorch tensor in memory](https://github.com/huggingface/transformers/blob/6303b5a7185fba43830db0cbb06c61861f57ddff/src/transformers/data/datasets/language_modeling.py#L88) instead of python list. We leave this in future work.

@LysandreJik @sgugger 

Comments:
ElectraTokenizerFast
Author: LysandreJik
Pull request: yes

Before the fix, when loading an `ElectraTokenizerFast`:

```py
from transformers import ElectraTokenizerFast

tokenizer = ElectraTokenizerFast.from_pretrained("ahotrod/electra_large_discriminator_squad2_512")
```
```
Traceback (most recent call last):
  File "/Users/jik/Library/Application Support/JetBrains/PyCharm2020.2/scratches/7735.py", line 3, in <module>
    tokenizer = ElectraTokenizerFast.from_pretrained("ahotrod/electra_large_discriminator_squad2_512")
  File "/Users/jik/Workspaces/python/transformers/src/transformers/tokenization_utils_base.py", line 1555, in from_pretrained
    resolved_vocab_files, pretrained_model_name_or_path, init_configuration, *init_inputs, **kwargs
  File "/Users/jik/Workspaces/python/transformers/src/transformers/tokenization_utils_base.py", line 1623, in _from_pretrained
    tokenizer = cls(*init_inputs, **init_kwargs)
  File "/Users/jik/Workspaces/python/transformers/src/transformers/tokenization_bert.py", line 641, in __init__
    **kwargs,
  File "/Users/jik/Workspaces/python/transformers/src/transformers/tokenization_utils_fast.py", line 89, in __init__
    self._tokenizer = convert_slow_tokenizer(slow_tokenizer)
  File "/Users/jik/Workspaces/python/transformers/src/transformers/convert_slow_tokenizer.py", line 565, in convert_slow_tokenizer
    converter_class = CONVERTERS[transformer_tokenizer.__class__.__name__]
KeyError: 'ElectraTokenizer'
```

Comments:
Bert base chinese model gives error :- EagerTensor object has no attribute 'size'
Author: akanyaani
Pull request: no

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->
     
- `transformers` version: 3.2.0
- Platform:
- Python version: 3.7
- PyTorch version (GPU?):
- Tensorflow version (GPU?): 3.2.1
- Using GPU in script?:
- Using distributed or parallel set-up in script?: No


## Information
I am just trying to get BERT embedding using the Chinese BERT base as explained in GitHub but getting an error.

Model I am using (Bert, XLNet ...): BERT (Chinese)


## To reproduce

```
from transformers import AutoTokenizer, AutoModel

tokenizer = AutoTokenizer.from_pretrained("bert-base-chinese")
model = AutoModel.from_pretrained("bert-base-chinese")

inputs = tokenizer("和 管理 ， 发挥 公路", return_tensors="tf")
outputs = model(**inputs)

```
Error 
```
AttributeError                            Traceback (most recent call last)
<ipython-input-30-481c0ebb1173> in <module>
      1 inputs = tokenizer("和 管理 ， 发挥 公路", return_tensors="tf")
      2 
----> 3 outputs = model(**inputs)

~/anaconda3/lib/python3.7/site-packages/torch/nn/modules/module.py in _call_impl(self, *input, **kwargs)
    720             result = self._slow_forward(*input, **kwargs)
    721         else:
--> 722             result = self.forward(*input, **kwargs)
    723         for hook in itertools.chain(
    724                 _global_forward_hooks.values(),

~/anaconda3/lib/python3.7/site-packages/transformers/modeling_bert.py in forward(self, input_ids, attention_mask, token_type_ids, position_ids, head_mask, inputs_embeds, encoder_hidden_states, encoder_attention_mask, output_attentions, output_hidden_states, return_dict)
    789             raise ValueError("You cannot specify both input_ids and inputs_embeds at the same time")
    790         elif input_ids is not None:
--> 791             input_shape = input_ids.size()
    792         elif inputs_embeds is not None:
    793             input_shape = inputs_embeds.size()[:-1]

AttributeError: 'tensorflow.python.framework.ops.EagerTensor' object has no attribute 'size'
```

Comments:
You are mixing the Pytorch and tf API. You should have "return_tensors="pt"" if you use PyTorch or use TFAutoModel for TensorFlow.

@esp32wrangler is correct!
Gpt1 for sequence classification
Author: fmcurti
Pull request: yes

# What does this PR do?

Adds sequence classification architecture for GPT-1,
Strongly based on modifications made in #7501

Fixes #7623 (issue) (Partially)


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dimiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests), 
      Pull Request section?
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to the it if that's the case.
- [x] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [x] Did you write any new necessary tests? 


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors which may be interested in your PR.

@LysandreJik Here is the new PR without that merge problem, let me know if there is anything that should be changed =)

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

 albert, bert, GPT2, XLM: @LysandreJik 
 tokenizers: @mfuntowicz
 Trainer: @sgugger
 Speed and Memory Benchmarks: @patrickvonplaten
 Model Cards: @julien-c
 Translation: @sshleifer
 Summarization: @sshleifer
 TextGeneration: @TevenLeScao 
 examples/distillation: @VictorSanh
 nlp datasets: [different repo](https://github.com/huggingface/nlp)
 rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)
 Text Generation: @TevenLeScao
 Blenderbot, Bart, Marian, Pegasus: @sshleifer
 T5: @patrickvonplaten
 Longformer/Reformer: @patrickvonplaten
 TransfoXL/XLNet: @TevenLeScao 
 examples/seq2seq: @sshleifer
 examples/bert-loses-patience: @JetRunner
 tensorflow: @jplu
 examples/token-classification: @stefan-it
 documentation: @sgugger
 -->

Comments:
This is great, thanks for working on it! There's something that I had forgotten in the initial GPT-2 implementation, which was to add it to the auto-models. I did it in this PR: https://github.com/huggingface/transformers/pull/7630.

Could you apply the fix to GPT-1 as well, before we merge?

> This is great, thanks for working on it! There's something that I had forgotten in the initial GPT-2 implementation, which was to add it to the auto-models. I did it in this PR: #7630.
> 
> Could you apply the fix to GPT-1 as well, before we merge?

Done! 😊

Thanks @fmcurti!
Make T5 Supports Gradient Checkpointing
Author: agemagician
Pull request: yes

# What does this PR do?

Since T5 3B and 11B models are really huge models to be fine-tuned on a single GPU, Gradient Checkpointing will allow this model to be fine-tuned on a single GPU but at the cost of more training time.

Fixes # (issue)


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dimiss the other checks if that's the case).
- [X] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests), 
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to the it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests? 


## Who can review?
 T5: @patrickvonplaten


Comments:
I've tested your code, it raise an error
`TypeError('CheckpointFunctionBackward.forward: expected Variable (got NoneType) for return value 1')                       
> /share/home/dwaydwaydway/miniconda3/envs/t5/lib/python3.7/site-packages/torch/utils/checkpoint.py(163)checkpoint()
    162 
--> 163     return CheckpointFunction.apply(function, preserve, *args)
    164`

@dwaydwaydway  yes, it is really annoying issue. I will try to fix it and open a new pull when it is done.
Adding Fast tokenizers for SentencePiece based tokenizers - Breaking: remove Transfo-XL fast tokenizer
Author: thomwolf
Pull request: yes

This pull request add the "fast" Rust tokenizer for the SentencePiece tokenizers as well.

Based on unreleased v0.9.0 of `tokenizers`.

Tokenizers:
- [x] Albert
- [x] Bart
- [x] Bert
- [x] Camembert
- [x] DistilBert
- [x] DPR
- [x] Electra
- [x] Funnel
- [x] GPT2
- [x] LongFormer
- [x] LXMert
- [x] MBart
- [x] MobileBert
- [x] OpenAI GPT
- [x] Pegasus
- [x] Reformer
- [x] RetriBert
- [x] Roberta
- [x] T5
- [x] XLM-Roberta
- [x] XLNet

Breaking:
- Fast version of Transformer-XL (which gave different tokenization results) is removed.

Remaining tokenizers without Fast implementations (no fast tokenizers expected in the short/mid-term):
- BertJapanese (special python libs for multi-linguality)
- CTRL (would require a specific BPE to handle missing merges)
- XLM (uses special python libs for multi-linguality)
- Flaubert (same as XLM)
- Transformer-XL (same as XLM)

Other fixes:
- Also allow to tokenizer Bert Japanese with Mecab token splitter and fix https://github.com/huggingface/datasets/issues/665
- deprecation warning in tokenizer methods are limited to one occurence per class instance

Comments:
Ready for review, the remaining failing tests should be ok after the next `tokenizers` RC release

Great job! I'm not entirely up to date with everything in transformers, but this looks very nice and clean!

Ok, yes I'll add documentation. We will probably wait to have a clean documentation in `tokenizers` as well so we can do proper cross-linking.
Fine tuned to Multi-choice dataset?
Author: Qzsl123
Pull request: no

Is it posible to fine tuned to the multi choices problems , which usually has one passage, question and ABCD four options? 

Comments:
Yes it is, the code is not written yet but I'm planning to work on it. The idea is to format the input data the same way the authors of [Improving Language Understanding with Unsupervised Learning](https://blog.openai.com/language-unsupervised/)


![Multiple choice GPT](https://i.imgur.com/z0Eanvy.png)

You run an inference `(context, choice)` for each choice, you compute the image of the `[CLS]` token by a linear layer with 1 output and then compute a softmax over the output of all choices.

I will try to create a PR with this code very soon. 

Thx for the reply.
Actually, I have the same plan. But I am not sure whether it will work. Anyway, I will have a try.

If it worked in the OpenAI paper, I don't really see why it wouldn't work with this architecture.

@Qzsl123 The code for multiple choice task is available in PR #96 if you want to test it.

@rodgzilla yeah, I am trying to run it. Thanks for the wonderful job!

> Yes it is, the code is not written yet but I'm planning to work on it. The idea is to format the input data the same way the authors of [Improving Language Understanding with Unsupervised Learning](https://blog.openai.com/language-unsupervised/)
> 
> ![Multiple choice GPT](https://camo.githubusercontent.com/e5d95abc42ca2acb493a710383c949eb01c10bfb/68747470733a2f2f692e696d6775722e636f6d2f7a3045616e76792e706e67)
> 
> You run an inference `(context, choice)` for each choice, you compute the image of the `[CLS]` token by a linear layer with 1 output and then compute a softmax over the output of all choices.
> 
> I will try to create a PR with this code very soon.

hi，The multi choices problem usually has one passage, question and ABCD four options。In your model, dose context means passage&question ?



Any update on this issue?
2 RAG test failures
Author: sshleifer
Pull request: no

The two failures here look related to recent tokenizer changes:

https://github.com/huggingface/transformers/runs/1236753957?check_suite_focus=true

```
=========================== short test summary info ============================

FAILED tests/test_modeling_rag.py::RagModelIntegrationTests::test_rag_sequence_generate_batch
FAILED tests/test_modeling_rag.py::RagModelIntegrationTests::test_rag_token_generate_batch
```

Comments:
@sshleifer - thanks a lot for the issue! 

This error seems related to https://github.com/huggingface/transformers/issues/7690#issuecomment-707382445.
Do not softmax when num_labels==1
Author: LysandreJik
Pull request: yes

When `config.num_labels=1`, the text classification pipeline shouldn't softmax over the results as it will always return 1. Instead, when the number of labels is 1, this will run a sigmoid over the result.

cf https://github.com/huggingface/transformers/issues/7493#issuecomment-706413792

Comments:
Adding optional trial argument to model_init
Author: madlag
Pull request: yes

# Model structure optimization 

This PR proposes to add the "trial" argument to the model_init function when using `trainer.hyperparameter_search`.
It's backward compatible using six to check the number of arguments of model_init through Python reflection.

```
def model_init(trial):
    if trial != None:
        layer_count = trial.suggest_int("layer_count", 2, 4)
    else:
        layer_count = 2
    return MyModel(layer_count)


trainer = Trainer(
   ...
    model_init=model_init,
   ...
)

trainer.hyperparameter_search(direction="maximize")
```



Comments:
from transformers import RagSequenceForGeneration gives ImportError
Author: mchari
Pull request: no

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->
     
- `transformers` version: 3.3.1
- Platform: Linux
- Python version:3.6.3
- PyTorch version (GPU?):
- Tensorflow version (GPU?):2.3.0
- Using GPU in script?:no
- Using distributed or parallel set-up in script?:no

### Who can help
@patrickvonplaten @sgugger 

## Information

Model I am using ():

The problem arises when using:
 


## To reproduce
1.python
>> from transformers import RagSequenceForGeneration           <- RagTokenizer is imported without any errors




## Expected behavior
I shouldn't get 
ImportError: cannot import name 'RagSequenceForGeneration'



Comments:
You don't have PyTorch installed in your environment. This is a PyTorch model.

Indeed , that was the problem.
The code in https://huggingface.co/transformers/model_doc/rag.html  has import torch after the "from transformers import RagSequenceForGeneration,..." statement, so I incorrectly concluded that torch is not needed for the import. Also, that RagTokenizer got imported without errors.
Thanks!
Tapas
Author: flozi00
Pull request: no
Labels: name: wontfix

name: New model

# 🌟 New model addition

## Model descriptin

Tapas extends Bert architecture and is a transformer-based Table QA model

Paper: https://arxiv.org/abs/2004.02349

## Open source status

* [X] the model implementation is available: (give details)
https://github.com/google-research/tapas
* [X] the model weights are available: (give details)
The pretrained weights and data for fine-tuning are linked in the repository readme

* [x] who are the authors:
@thomasmueller-google is one of the authors

Comments:
hey, @thomwolf this would be a great addition! have a look into it.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


I think @NielsRogge is working on this over here: https://github.com/NielsRogge/transformers/tree/modeling_tapas
Fix bert position ids in DPR convert script
Author: lhoestq
Pull request: yes

https://github.com/huggingface/transformers/commit/614fef1691edb806de976756d4948ecbcd0c0ca3 introduced buffers for position ids for BERT that breaks the DPR convert script since the DPR weights don't have those.

To fix that I followed @LysandreJik 's suggestion to manually add the position ids to the state dict before loading it into the model.

Comments:
Update of DialoGPT `max_length`
Author: guillaume-be
Pull request: no

### Who can help
@patrickvonplaten 

## Information

Following https://github.com/huggingface/transformers/pull/5516, the DialoGPT models `max_length` has not been updated, and defaults to the `generate` value of 20. This value is very low for a conversational pipeline and would lead to answers that completely ignore the history (truncation happens to ensure enough space is available for the response).

## Expected behavior

- The configuration files (e.g. https://s3.amazonaws.com/models.huggingface.co/bert/microsoft/DialoGPT-medium/config.json) need to store a `max_length` equal to 1000.
- If the suggested config file structure from the PR is adopted, the code should be updated to read this value instead
```json
  "task_specific_params" : {
      "dialogue": { 
          "max_length": 1000
      }
  }
```


Comments:
Very true! Thanks for the notification - will upload all DialoGPT configs.

Done - note however that the "task" name was `conversational` not `dialogue`
Is there any way to control the input of a layer of `Longformer`?
Author: h56cho
Pull request: no

Hello,

Is there any way that I can directly control the input to a layer of the `Longformer` model, similar to `GPT2.transformer.h[]`?

I tried `best_model_longformer.longformer.encoder.layer[layer_index](input_hidden_state_for_layer)` but it's giving this error:

```python
Traceback (most recent call last):
  File "SEED_125_V20_15_LONGFORMER.py", line 426, in <module>
    main_function('/home/ec2-user/G1G2.txt','/home/ec2-user/G1G2_answer_num.txt', num_iter)
  File "SEED_125_V20_15_LONGFORMER.py", line 388, in main_function
    best_model_longformer)
  File "SEED_125_V20_15_LONGFORMER.py", line 205, in fill_MC_loss_accuracy_tensor
    best_model_longformer.longformer.encoder.layer[j](input_hidden_state)
  File "/home/ec2-user/anaconda3/lib/python3.7/site-packages/torch/nn/modules/module.py", line 722, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/home/ec2-user/anaconda3/lib/python3.7/site-packages/transformers/modeling_longformer.py", line 852, in forward
    output_attentions=output_attentions,
  File "/home/ec2-user/anaconda3/lib/python3.7/site-packages/torch/nn/modules/module.py", line 722, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/home/ec2-user/anaconda3/lib/python3.7/site-packages/transformers/modeling_longformer.py", line 796, in forward
    output_attentions,
  File "/home/ec2-user/anaconda3/lib/python3.7/site-packages/torch/nn/modules/module.py", line 722, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/home/ec2-user/anaconda3/lib/python3.7/site-packages/transformers/modeling_longformer.py", line 241, in forward
    attention_mask = attention_mask.squeeze(dim=2).squeeze(dim=1)
AttributeError: 'NoneType' object has no attribute 'squeeze'
```
:S thank you,

Comments:
rag examples tests fail
Author: sshleifer
Pull request: no

```
================================================================ ERRORS =================================================================
______________________________________ ERROR collecting examples/rag/test_distributed_retriever.py ______________________________________
ImportError while importing test module '/Users/shleifer/transformers_fork/examples/rag/test_distributed_retriever.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
examples/rag/test_distributed_retriever.py:26: in <module>
    from examples.rag.distributed_retriever import RagPyTorchDistributedRetriever  # noqa: E402 # isort:skip
E   ModuleNotFoundError: No module named 'examples.rag'
```

Comments: