noamaanMulla-03/transformers-issues huggingface dataset:
Load from tf2.0 checkpoint fail
Author: stevewyl
Pull request: no
Labels: name: wontfix

# 🐛 Bug

## Information

Model I am using (Bert, XLNet ...): Bert

Language I am using the model on (English, Chinese ...): English

The problem arises when using:
* [x] the official example scripts: (give details below)
* [ ] my own modified scripts: (give details below)


The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [x] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:

1. Download tf2.0 checkpoint from https://storage.googleapis.com/cloud-tpu-checkpoints/bert/keras_bert/uncased_L-12_H-768_A-12.tar.gz
2. unpack the model tar.gz to `bert_models` folder
3. start an iPython console and type following codes: 
```python
import tensorflow as tf
from transformers import TFBertModel, BertConfig
config = BertConfig.from_json_file("./bert_models/uncased_L-12_H-768_A-12/bert_config.json")
model = TFBertModel.from_pretrained("./bert_models/uncased_L-12_H-768_A-12/bert_model.ckpt.index", config=config)
```
4. I check the original code from tf2.0 and found they didn't implement model.load_weights when by_name is True. Error is following:
NotImplementedError: Weights may only be loaded based on topology into Models when loading TensorFlow-formatted weights (got by_name=True to load_weights)

<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

## Expected behavior

<!-- A clear and concise description of what you would expect to happen. -->

## Environment

* OS: CentOS Linux release 7.4.1708 (Core)
* Python version: 3.7.6
* PyTorch version: 1.3.1
* `transformers` version (or branch):
* Using GPU ? Yes
* Distributed or parallel setup ? No
* Any other relevant information:


Comments:
Hi, in order to convert an official checkpoint to a checkpoint readable by `transformers`, you need to use the script `convert_bert_original_tf_checkpoint_to_pytorch`. You can then load it in a `BertModel` (PyTorch) or a `TFBertModel` (TensorFlow), by specifying the argument `from_pt=True` in your `from_pretrained` method.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
BertSelfAttention have not Add and Norm layer, why???
Author: weiliangxiao
Pull request: no


Comments:
It's in [`BertSelfOutput`](https://github.com/huggingface/transformers/blob/master/src/transformers/modeling_bert.py#L261) which is called right after the `BertSelfAttention` in [`BertAttention`](https://github.com/huggingface/transformers/blob/master/src/transformers/modeling_bert.py#L316).
❓Adding new tokens to pre-trained tokenizer
Author: yash1994
Pull request: no

## Details
Hi, I am working with DistilBERT multilingual model for sequence classification tasks where I need to add some additional languages apart from mentioned [here](https://github.com/google-research/bert/blob/master/multilingual.md#list-of-languages). And for that, I am struggling to find the correct way to update tokenizer. From the documentation, I inferred that first, i have to get all new tokens in a list, call `tokenizer.add_tokens()` and then again i have to pass those new sentences to tokenizer and get them tokenized. So the real question: is there any method which i use to update tokenizer and tokenize sentence at the same time (when tokenizer sees unknown token it adds the token to the dictionary). Thanks in advance.

Comments:
docs are pretty nice imho;

If I had found what I looking for in the documentation then why would I open an issue and waste someone else's time? I know other approaches to this problem but one seemed to be more time saving so just checking for implementation available. But now it seems that people are more concerned about critical bugs only in the issue section. Closing the issue for good.

There is no way to dynamically add unknown tokens to the vocabulary. The simplest way to do it would be to encode the sequence, detect unknowns, and then add these to the vocabulary, which seems to be what you did!

Please be aware that you will have to resize the model's embedding matrix according to the tokens you've added.

Or you can map all such tokens a group of them maybe to an OOV kinda token as well. 

So the only way to update tokenizer is to get all unknowns first then resize model embedding matrix. Thanks @LysandreJik and @AdityaSoni19031997
Regarding distlbert uncased model's size
Author: divyag11
Pull request: no
Labels: name: wontfix

I finetuned the distlbert uncased model, and I had a thought that since it is a lower layer model, it should have less weight. But, to my surprise, I find that the model generated after finetuning, I saved it as :
      tf.saved_model.save(model, "./tempdir/distilbert/2/")
and the tf model got saved.
this model has a very high size(of 810 Mb), although it should be less .And the original bert mdoel which is a large model has a size of 410 Mb.
Please look into the matter

Comments:
please reply

You're comparing two formats that are different, so the comparison doesn't really make sense. The BERT model weighs 410MB when saved as a HDF5 file, whereas DistilBERT weighs 810MB when saved as a SavedModel, which also contains the graph and variables.

Saving both files in HDF5:

```py
from transformers import TFBertModel, TFDistilBertModel

bert = TFBertModel.from_pretrained("bert-base-uncased")
distilbert = TFDistilBert.from_pretrained("distilbert-base-uncased")

bert.save_pretrained("bert")
distilbert.savePretrained("distilbert")
```

`ls` in "bert" -> 414MB for the `tf_model.h5`
`ls` in "distilbert" -> 254MB for the `tf_model.h5`

I got your answer, that's correct.But, since I need to serve the model  using tfserving,so I need a SavedModel format only.
1)Or is there any way to serve   .h5 models through tfserving?
2)Or is there any way i can remove adam momentum and such trainig variables from my model to reduce the size in tensorflow 2.1?,because i have seen in tf 2.x freezing graph is deprecated


This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Added README huseinzol05/albert-tiny-bahasa-cased
Author: huseinzol05
Pull request: yes
Labels: name: model card


Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/3746?src=pr&el=h1) Report
> Merging [#3746](https://codecov.io/gh/huggingface/transformers/pull/3746?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/700ccf6e35616fcbee59de81edd60cec9e14fb6b&el=desc) will **increase** coverage by `0.01%`.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/3746/graphs/tree.svg?width=650&height=150&src=pr&token=9qOlN6Hb1c)](https://codecov.io/gh/huggingface/transformers/pull/3746?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master    #3746      +/-   ##
==========================================
+ Coverage   78.26%   78.27%   +0.01%     
==========================================
  Files         106      106              
  Lines       17928    17928              
==========================================
+ Hits        14031    14033       +2     
+ Misses       3897     3895       -2     
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/3746?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [src/transformers/modeling\_tf\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/3746/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ190Zl91dGlscy5weQ==) | `93.12% <0.00%> (+0.32%)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/3746?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/3746?src=pr&el=footer). Last update [700ccf6...65d2323](https://codecov.io/gh/huggingface/transformers/pull/3746?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


Looks good! [model page](https://huggingface.co/huseinzol05/tiny-bert-bahasa-cased)
Mismatch of loss shape in document with output of TransfoXL
Author: TobiasLee
Pull request: no

# 🐛 Bug

## Information

Model: Transformer-XL

Language: English

The problem arises when using: a small demo of transformer-xl output

## To reproduce

Steps to reproduce the behavior:
```python3
model_name = 'transfo-xl-wt103'
model = AutoModelWithLMHead.from_pretrained(model_name)
tokenizer = AutoTokenizer.from_pretrained(model_name)
sentence = 'Hello world'
tokenize_input = tokenizer.tokenize(sentence) 
tensor_input = torch.tensor([tokenizer.convert_tokens_to_ids(tokenize_input)])
outputs = model(tensor_input, labels=tensor_input)
print(outputs[0].size())
```
run the code above 


## Expected behavior

The  output[0] is supposed to be a  tensor shaped as `(1,)`, as described in the [source code](https://github.com/huggingface/transformers/blob/master/src/transformers/modeling_transfo_xl.py#L862), however, the actual shape is (1, 2)  (bsz, sequence_len).

## Environment info
<!-- You can run the command `python transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->
     
- `transformers` version: 2.7.0
- Platform: Ubuntu 18.04
- Python version: 3.6
- PyTorch version (GPU?): 1.4.0, with 1080Ti 
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: No 


Comments:
The output shape looks correct to me 
"loss (:obj:`torch.FloatTensor` of shape `(batch_size, sequence_length)`" means that 
`outputs[0]` should be of shape ` [batch_size, sequence_length] ` and it is ` [1, 2] `, so that is correct no?

yes, since the document was updated via #3661
Masked LM and TFBertForSequenceClassification
Author: tomerwul
Pull request: no
Labels: name: wontfix

Hello,

Is it correct to say that fine-tuning a TFBertForSequenceClassification model is the same as fine-tuning BERT's mlm and in additional a classification layer at the same time?

Thanks!

Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Quick question difference output of Bert models compared to Electra
Author: Stuffooh
Pull request: no

Hi everyone,

I am a little confused at the moment.

When I run `outputs = model.roberta(input_ids, attention_mask)` or `model.albert(input_ids, attention_mask)` the length of output is 3 and looks like this:

`output[0].shape = [4, 249, 768]`
`output[1][0].shape = [768]`
`output[2][0].shape = [4,249,768]`

When I run model.electra(input_ids, attention_mask)` the length of output is 2 and looks like this:

`output[0].shape = [4, 251, 768]`
`output[1][0].shape = [4, 251, 768]`

I checked the config files of both models and both `output hidden states` etc. seems to be set to False and in the code I don't specify to output anything extra for either model.

Can someone explain why Electra all of a sudden outputs less compared to other models and also what output[0], output[1] and output[2] mean for Bert and for Electra?

I checked the documentation but there it states all the output except scores is optional so I am confused what output contains know since to my understanding I haven't specified to output and of the optional output.

Thanks in advance for helping me clear this confusion up.


Comments:
For questions like this it's often best to dive into the source code. It helps understanding it a lot.

But your example seems wrong, you may have made some copy-pasting errors. If `output[1][0].shape = [768]` then also, because tensors across an axis must have same dimensions (like matrices) `output[2][0].shape = [768]`.

@BramVanroy I double checked but it is not a copy-pasting error as far as I can tell. I will dive into the source code to try to understand why the length of electra's output is different from the bert models.

Can you post a reproducible full, but minimal, example?

@BramVanroy I figured it out. Turns out even though in the config the output_hidden_states is set to false somewhere hidden in the code I am using it get sets to true.

In case of the bert models output[2] are the hidden layers and for electra [1] are the hidden layers. I'm still not very sure what output[1] is in case of the output of the bert models but for my particular use case it is not important right now.

Thank you for taking your time to help me. I will close the issue.
uss
Author: hahadashi
Pull request: no

# ❓ Questions & Help

<!-- The GitHub issue tracker is primarly intended for bugs, feature requests,
     new models and benchmarks, and migration questions. For all other questions,
     we direct you to Stack Overflow (SO) where a whole community of PyTorch and
     Tensorflow enthusiast can help you out. Make sure to tag your question with the
     right deep learning framework as well as the huggingface-transformers tag: 
     https://stackoverflow.com/questions/tagged/huggingface-transformers 
     
     If your question wasn't answered after a period of time on Stack Overflow, you
     can always open a question on GitHub. You should then link to the SO question 
     that you posted.
     -->

## Details
<!-- Description of your issue -->

<!-- You should first ask your question on SO, and only if
     you didn't get an answer ask it here on GitHub. -->
**A link to original question on Stack Overflow**: 

Comments:
Was this opened by mistake? Can we close it?
ImportError: cannot import name 'GradientAccumulator'
Author: DenceChen
Pull request: no
Labels: name: wontfix

name: Version mismatch

transformers==2.4.1;
tensoflow==2.1.0;
torch==1.4.0;
when i start with follow code get some error and i didn't have tensoflow-gpu.

**code**
---------------------------------------------------------------------------
from transformers import (
    TF2_WEIGHTS_NAME,
    BertConfig,
    BertTokenizer,
    DistilBertConfig,
    DistilBertTokenizer,
    GradientAccumulator,
    RobertaConfig,
    RobertaTokenizer,
    TFBertForTokenClassification,
    TFDistilBertForTokenClassification,
    TFRobertaForTokenClassification,
    create_optimizer,
)


**error**
---------------------------------------------------------------------------
ImportError                               Traceback (most recent call last)
<ipython-input-2-4dd6bfff15e9> in <module>()
     12 from seqeval import metrics
     13 
---> 14 from transformers import (
     15     TF2_WEIGHTS_NAME,
     16     BertConfig,

ImportError: cannot import name 'GradientAccumulator'


Comments:
Hi,
anyone found workaround for this issue?
Thanks

This shouldn't happen with transformers v2.4.1 and tensorflow >= 2.0.0.

I can't replicate this issue with the versions you mentioned.

Would you mind telling me what gets printed out when you run the following snippet?

```py
from transformers import __version__
import tensorflow as tf

print("Transformers version", __version__)
print("TensorFlow version", tf.__version__)
```

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


well done
Why is the seq_len dimension hard coded to be the first dimension of BERT's input?
Author: entslscheia
Pull request: no

According to your [code](https://github.com/huggingface/transformers/blob/2e81b9d8d76a4d41a13f74eb5e0f4a65d8143cab/src/transformers/modeling_bert.py#L164), the `seq_len` dimension always corresponds to the first dimension of the input tensor. I found this problem when I tried to feed a tensor of shape `(batch_size, num_seq, seq_len)` and error occurred. Is there any specific reason for you to adopt this setting? Given that normally a PyTorch` Embedding` layer  can take an input of any shape, it doesn't look so natural to me to have this setting. Any suggestions? Many thanks!

Comments:
My wild guess is maybe there is no specific reason for this. It's just an implementation issue. If I really want do feed a tensor of different number of dimensions I can just do reshape twice (i.e., one before forward one after).
[CamemBERT] Add CamembertForQuestionAnswering
Author: alekseiancheruk
Pull request: no
Labels: name: wontfix

Firstly, a huge thanks to Hugging Face team for their great work ! As we have now Camembert, it would be nice to use it for question answering using transformers ! You can find SQuAD in French on GitHub so it would be so easy to a lot of people to fine-tune Camembert for this task. 

Please consider it in future releases 😉


Comments:
I wonder if loading as roberta really works for camembert

What do you mean with this statement?

> I wonder if loading as roberta really works for camembert

Happy to review a PR for this (should be pretty easy to add!)

> 
> 
> What do you mean with this statement?
> 
> > I wonder if loading as roberta really works for camembert

you can load camembert config and checkpoints as roberta models but I'm not perfectly sure it's identical. 
I did'nt checked if there was RobertaForQuestionAnswering so my comment is partly irrevelent. However both slould be added if possible. maybe some generic heads using PreTrainedModel could be possible even in not efficient

> > What do you mean with this statement?
> > > I wonder if loading as roberta really works for camembert
> 
> you can load camembert config and checkpoints as roberta models but I'm not perfectly sure it's identical.
> I did'nt checked if there was RobertaForQuestionAnswering so my comment is partly irrevelent. However both slould be added if possible. maybe some generic heads using PreTrainedModel could be possible even in not efficient

At the moment, you **can't** use RoBERTa model for QuestionAnswering. You can use `RoBERTa` model for **token classification**, **multiple choice**, **sequence classification** and **MaskedLM** (with the usual `RobertaFor*` naming convention).

Hi, thanks to the Hugging Face team for the amazing work ! I think there is a PR here to add camembert for question answering : https://github.com/huggingface/transformers/pull/2746

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Why the first item of the config.json of bert is "architectures": ["BertForMaskedLM"]
Author: janyChan
Pull request: no

{
  "architectures": [
    "BertForMaskedLM"
  ],
  "attention_probs_dropout_prob": 0.1, 
  "directionality": "bidi", 
  "hidden_act": "gelu", 
  "hidden_dropout_prob": 0.1, 
  "hidden_size": 768, 
  "initializer_range": 0.02, 
  "intermediate_size": 3072, 
  "max_position_embeddings": 512, 
  "num_attention_heads": 12, 
  "num_hidden_layers": 12, 
  "pooler_fc_size": 768, 
  "pooler_num_attention_heads": 12, 
  "pooler_num_fc_layers": 3, 
  "pooler_size_per_head": 128, 
  "pooler_type": "first_token_transform", 
  "type_vocab_size": 2, 
  "vocab_size": 21128
}

why the first item is "architectures": [
    "BertForMaskedLM"
  ]
I see, this is not available in the google version.
Could you tell me what does it mean？

Comments:
This is a property we added to be able to know what kind of final layers the model has. It is used for instance to enable tagging and filtering on our model hub: https://huggingface.co/models
[WIP] EncoderDecoder model that works
Author: sshleifer
Pull request: yes

Continuing #3383 from @patrickvonplaten to facilitate MarianMT project.

### Targeted API:
```python
model = EncoderDecoderModel.from_model_names('bert-base-uncased', 'bert-based-uncased')
model.save_pretrained('bert2bert')
model.from_pretrained('bert2bert') # way 2
```

### TODO
- support test_common 
- test forward, generate
- raise useful errors for incompatible encoder/decoder combinations




Comments:
TransfoXLLMHead doesn't shift labels internally when called for loss
Author: TevenLeScao
Pull request: no

# 🐛 Bug

When called with labels to get the language-modeling loss, `TransfoXLLMHead.forward` computes the NLLLoss of the outputs directly against the labels, rather than against the shifted labels like the documentation indicates (and like the other models). This makes it impossible to train with `lm_labels = input_ids` as suggested by the doc.

## Information

Model I am using: TransformerXL

Language I am using the model on: English

The problem arises when using:
* [x] my own modified scripts:

The task I am working on is:
* [x] my own task or dataset:

## To reproduce

```
import torch
from transformers import TransfoXLConfig, TransfoXLLMHeadModel

config = TransfoXLConfig()
lm = TransfoXLLMHeadModel(config)
test_tensor = torch.LongTensor([[0]])
print(lm(input_ids=test_tensor, labels=test_tensor)[0])
```

A 1x1 loss tensor is returned.

## Expected behavior

As there is only 1 token in the input tensor, no loss should be returned: there's no next label to compare the output against. For example, running this with GPT2

```
import torch
from transformers import GPT2Config, GPT2LMHeadModel

config = GPT2Config()
lm = GPT2LMHeadModel(config)
test_tensor = torch.LongTensor([[0]])
print(lm(input_ids=test_tensor, labels=test_tensor)[0])
```

returns `tensor(nan, grad_fn=<NllLossBackward>)`.

## Environment info
     
- `transformers` version: 2.8.0
- Platform: Linux-5.3.0-45-generic-x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.6.9
- PyTorch version (GPU?): 1.4.0 (True)
- Tensorflow version (GPU?): not installed (NA)
- Using GPU in script?: False
- Using distributed or parallel set-up in script?: False



Comments: