noamaanMulla-03/transformers-issues huggingface dataset:
Question: Token sequence length longer maximum sequence length
Author: CMobley7
Pull request: no
Labels: name: wontfix

## ❓ Questions & Help

I'm using `run_glue.py` with a task name of `SST-2` to fine-tune a binary classifier on my data, which I put into the required format. However, some of my data's sentences are longer than the `max_seq_length` of `512` for `BERT` and `RoBERTa`; so, I get

`WARNING - transformers.tokenization_utils -   Token indices sequence length is longer than the specified maximum sequence length for this model (length_of_my_string > 512). Running this sequence through the model will result in indexing errors`.

What exactly is happening here? Are the training examples with more than `510` tokens still being used? If so, is the string being truncated down to `[CLS]` + the `first 510 tokens` + `[SEP]`? Is there any way to increase the `max_seq_length` or implement something like `head+tail`, which selects the `first 128` and the `last 382` tokens like suggested in this [paper](https://arxiv.org/pdf/1905.05583.pdf). That paper also uses `discriminative learning rate` as suggested [here](https://arxiv.org/pdf/1801.06146.pdf). Is there any plan to implement this?



Comments:
Going through the source code, the sequence is actually truncated.

https://github.com/huggingface/transformers/blob/079bfb32fba4f2b39d344ca7af88d79a3ff27c7c/transformers/tokenization_utils.py#L846-L853

The warning occurs because `encode_plus` calls `convert_tokens_to_ids` _first_ and only then the IDs are truncated. The warning originates from `convert_tokens_to_ids` before truncation has happened. This is quite confusing indeed, since in the end result the IDs _are_ truncated.

Perhaps one of the maintainers can chip in.

> Going through the source code, the sequence is actually truncated.
> 
> https://github.com/huggingface/transformers/blob/079bfb32fba4f2b39d344ca7af88d79a3ff27c7c/transformers/tokenization_utils.py#L846-L853
> 
> The warning occurs because `encode_plus` calls `convert_tokens_to_ids` _first_ and only then the IDs are truncated. The warning originates from `convert_tokens_to_ids` before truncation has happened. This is quite confusing indeed, since in the end result the IDs _are_ truncated.
> 
> Perhaps one of the maintainers can chip in.

So, it safe to use or not?

This should have been patched in release 2.2.0.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Edit a way to get `projected_context_layer`
Author: jinkilee
Pull request: yes

I edited a way to get `projected_context_layer`.

Instead of doing
'''
w = (
    self.dense.weight.t()
    .view(self.num_attention_heads, self.attention_head_size, self.hidden_size)
    .to(context_layer.dtype)
)
b = self.dense.bias.to(context_layer.dtype)
projected_context_layer = torch.einsum("bfnd,ndh->bfh", context_layer, w) + b
'''

I added `self.merge_last_ndims` at `AlbertAttention`.
'''
def merge_last_ndims(self, x, n_dims):
    s = x.size()
    assert n_dims > 1 and n_dims < len(s)
    return x.view(*s[:-n_dims], -1)
'''

I commited this one yesterday, but that one didn't pass the tests, so I re-wrote my code and re-committing now.

Comments:
Hi, please check the [contribution guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests) for the code quality tests to pass.

Why did you edit this, does this solve a bug or add new functionality?

Hi i edited this because i read a comment like..
“ # Should find a better way to do this “
Not because of bug or new features

# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/2602?src=pr&el=h1) Report
> Merging [#2602](https://codecov.io/gh/huggingface/transformers/pull/2602?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/23c6998bf46e43092fc59543ea7795074a720f08?src=pr&el=desc) will **increase** coverage by `<.01%`.
> The diff coverage is `100%`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/2602/graphs/tree.svg?width=650&token=9qOlN6Hb1c&height=150&src=pr)](https://codecov.io/gh/huggingface/transformers/pull/2602?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master    #2602      +/-   ##
==========================================
+ Coverage   74.61%   74.61%   +<.01%     
==========================================
  Files          87       87              
  Lines       14802    14804       +2     
==========================================
+ Hits        11044    11046       +2     
  Misses       3758     3758
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/2602?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [src/transformers/modeling\_albert.py](https://codecov.io/gh/huggingface/transformers/pull/2602/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ19hbGJlcnQucHk=) | `79.03% <100%> (+0.11%)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/2602?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/2602?src=pr&el=footer). Last update [23c6998...a96e39e](https://codecov.io/gh/huggingface/transformers/pull/2602?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
How to persist cloud-based transformers
Author: jmwoloso
Pull request: no
Labels: name: wontfix

## ❓ Questions & Help

I'm using this repo in the cloud and attempting to persist the model fails as it seems HDF5 and thus h5py doesn't support that paradigm per https://github.com/h5py/h5py/issues/925

What is the recommended method of saving the model in this scenario? Thanks!


Comments:
Are you trying to serialize which type of model to .hdf5 file? A pre-trained model such as BertFor* or a custom model trained with Transformers? Are you using GCP or Azure or AWS?

> ##  Questions & Help
> I'm using this repo in the cloud and attempting to persist the model fails as it seems HDF5 and thus h5py doesn't support that paradigm per [h5py/h5py#925](https://github.com/h5py/h5py/issues/925)
> 
> What is the recommended method of saving the model in this scenario? Thanks!

Hi @TheEdoardo93. I'm trying to serialize TFBertFor* that has been fine-tuned. I'm on Azure.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Is transformers ovewriting tokenizer?
Author: diogocortiz
Pull request: no

Hello. I haven't been able to use tokenizer since friday. 
It seems that if I install transformers via pip it overwrites tokenizer installation with a version that doesn't work. 

If I get a new instance and do that:

`pip install transformers` 

When, I do that:
`pip install tokenizers`

I got the following msg:

> Requirement` already satisfied: tokenizers in /usr/local/lib/python3.7/site-packages (0.0.11)

And if I tried to import, I got this error msg:

> ImportError: cannot import name 'BertWordPieceTokenizer' from 'tokenizers'

I was wondering if it is a problem related to the new Transformers you released last Friday.


Comments:
Hi, I believe your issue was solved with https://github.com/huggingface/tokenizers/issues/120

Sure. I will close this case.
glue.py: AttributeError: 'numpy.str_' object has no attribute 'text_a'
Author: pacebrian0
Pull request: no

when I am executing the glue data conversion i.e.

`sequences = glue_convert_examples_to_features(X_train, tokenizer, max_length=MAX_SEQUENCE_LENGTH, task='mrpc')
`
I'm getting this error:

> I0126 11:57:07.862119 16252 glue.py:70] Using label list ['0', '1'] for task mrpc
> I0126 11:57:07.863118 16252 glue.py:73] Using output mode classification for task mrpc
> I0126 11:57:07.864120 16252 glue.py:80] Writing example 0
> ---------------------------------------------------------------------------
> 
> AttributeError                            Traceback (most recent call last)
> 
> <ipython-input-11-621d8071aa9a> in <module>
>       2 #test_sequences = [tokenizer.encode(xxt,add_special_tokens=True) for xxt in X_test]
>       3 #val_sequences = [tokenizer.encode(xxt,add_special_tokens=True) for xxt in X_val]
> ----> 4 sequences = glue_convert_examples_to_features(df['cleantext'], tokenizer, max_length=MAX_SEQUENCE_LENGTH, task='mrpc')
>       5 val_sequences = glue_convert_examples_to_features(X_val, tokenizer, max_length=MAX_SEQUENCE_LENGTH, task='mrpc')
>       6 test_sequences = glue_convert_examples_to_features(X_val, tokenizer, max_length=MAX_SEQUENCE_LENGTH, task='mrpc')
> 
> d:\anaconda3\envs\t2\lib\site-packages\transformers\data\processors\glue.py in glue_convert_examples_to_features(examples, tokenizer, max_length, task, label_list, output_mode, pad_on_left, pad_token, pad_token_segment_id, mask_padding_with_zero)
>      84 
>      85         inputs = tokenizer.encode_plus(
> ---> 86             example.text_a,
>      87             example.text_b,
>      88             add_special_tokens=True,
> 
> AttributeError: 'str' object has no attribute 'text_a'

As tokenizer I'm using 
`tokenizer = BertTokenizer.from_pretrained('bert-base-cased')`

and my numpy version is 1.18.1, tensorflow version 2.1.0 (base 2.1.0), transformers version 2.3.0


Comments:
I think the problem was due to the dataset not being set to (index,example) structure

@pacebrian0  Could you post what changes did you make?

I decided to use simpletransformers python package, which allows you to train custom datasets.
The above problem can only be solved by using tensorflow-datasets data as far as I know

Ah! The same as me. I am using that same package, but had no idea that those problems could be solved only with tensorflow-datasets.
Worse F1 on squad2 with finetune+distil distilroberta-base than just finetune
Author: volker42maru
Pull request: no
Labels: name: wontfix

Hi there, 
I am trying to finetune distilroberta on squad2. First, I simply used the _distilroberta-base_ model and finetuned it on the squad2 dataset using `run_squad.py`, which gave me **74/71 F1/EM**. It's a lot worse than the roberta-base accuracy.

Currently, I am trying to finetune+distil (from roberta-base squad2 finetuned model) using `run_squad_w_distillation.py`. My roberta-base squad2 finetuned model has around 83/80 F1/EM. However, when I try to finetune+distil _distilroberta-base_ with the finetuned roberta-base as teacher, I only get around **63/60 F1/EM**. Maybe my hyperparams are way off or I need to train longer? Here's my current config:
- learning_rate=3e-5
- total_batch_size=16
- num_train_epochs=2
- max_seq_length=384

I left all other hyperparams as default.

I also checked out some predictions and it seems the model most of the time predicts _no answer_ as the best answer. In cases where it actually predicts an answer, the accuracy is not that bad.

Would be awesome to get some feedback on this, as I am trying to do inference on CPU and a distilled model would greatly benefit me in this case.

Cheers

Comments:
cc @VictorSanh 

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Possible error in the HuggingFace Transformers documentation?
Author: h56cho
Pull request: no
Labels: name: wontfix

Hello,

According to HuggingFace Transformers documentation website (https://huggingface.co/transformers/model_doc/gpt2.html#gpt2doubleheadsmodel), under the GPT2DoubleHeadsModel, it defines the output lm_prediction_scores as the following:

`lm_prediction_scores: torch.FloatTensor of shape (batch_size, num_choices, sequence_length, config.vocab_size)`

To me this doesn't make sense. Shouldn't the dimension of `lm_prediction_scores` for `GPT2DoubleHeadsModel` be just `(batch_size, sequence_length, config.vocab_size)`? [no `num_choices` in the middle]

Thank you,


Comments:
The documentation seems correct to me. Have you tried out the example code that's been provided? If you run it and check the resulting `lm_prediction_scores`, you'll see its shape is `torch.Size([1, 2, 7, 50258])`, 2 being the length of `choices`.

This comment, and the linked blog post, explains the *DoubleHeadsModel pretty well:
https://github.com/huggingface/transformers/issues/1794#issuecomment-552627190

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Remove redundant hidden states
Author: LysandreJik
Pull request: yes

The quickstart showcasing the usage of the Model2Model currently fails. This is due to a positional argument that should be a named argument.

As I understand it, the `encoder_hidden_states` are already present in the `kwargs_decoder` dictionary, there is therefore no need to pass it to the decoder forward call.

With the current quickstart example this crashes as the position of the `encoder_hidden_states` means it's passed as an `attention_mask`.

Please correct me if I'm wrong @rlouf @thomwolf 

Comments:
Fine tuning XLMRoberta for Question Answering
Author: houdaM97
Pull request: no

# ❓ Questions & Help

<!-- The GitHub issue tracker is primarly intended for bugs, feature requests,
     new models and benchmarks, and migration questions. For all other questions,
     we direct you to Stack Overflow (SO) where a whole community of PyTorch and
     Tensorflow enthusiast can help you out. Make sure to tag your question with the
     right deep learning framework as well as the huggingface-transformers tag: 
     https://stackoverflow.com/questions/tagged/huggingface-transformers 
     
     If your question wasn't answered after a period of time on Stack Overflow, you
     can always open a question on GitHub. You should then link to the SO question 
     that you posted.
     -->

## Details
<!-- Description of your issue -->

<!-- You should first ask your question on SO, and only if
     you didn't get an answer ask it here on GitHub. -->
**A link to original question on Stack Overflow**: 

I'm trying to fine tune XLM Roberta for Question Answering in tensorflow version, and the question is do i need to convert the pytorch pretrained model to tensorflow because 'xlm-roberta-case' = "https://s3.amazonaws.com/models.huggingface.co/bert/xlm-roberta-base-pytorch_model.bin", if so how can i do it? i tried to use load_pytorch_model_in_tf2_model() but i had errors!!
 thank you for help in advance.


Comments:
Hi, we don't currently have an implementation of XLM RoBERTa in tensorflow.

I guess this one can now be closed since TensorFlow XLM-RoBERTa was released with 2.4.0. Thanks @LysandreJik @jplu . Quick question though: I guess you are not retraining the LM but convert the pytorch weighs. Is there any script in huggingface to do this?

Right now the easiest way to convert the PyTorch weights to TensorFlow when the two implementations are in huggingface/transformers is the following, for e.g. XLM-R:

```py
from transformers import XLMRobertaModel, TFXLMRobertaModel

pytorch_model = XLMRobertaModel.from_pretrained("xlm-roberta-base")  # Checkpoint on S3
pytorch_model.save_pretrained("pytorch_checkpoint_directory")  # Save it to a directory

tensorflow_model = TFXLMRobertaModel.from_pretrained("pytorch_checkpoint_directory", from_pt=True)  # Load from directory in TF
```

You can then save that TensorFlow model using the `save_pretrained` method, and you can do it the other way around too to conver TensorFlow models to PyTorch models.

@houdaM97, as @nchocho said XLM-R in TensorFlow was released in v2.4.0 last week. There are no official checkpoints on our s3 however, but there are contributed community checkpoints from @jplu you can use instead:

```py
from transformers import TFXLMRobertaModel

model = TFXLMRobertaModel.from_pretrained("jplu/tf-xlm-roberta-base")
```
BertForTokenClassification for NER . what is the conclusion of  this output ?
Author: AjitAntony
Pull request: no
Labels: name: wontfix

## ❓ Questions & Help

<!-- A clear and concise description of the question. -->
Hi ,
Im trying to perform NER using BertForTokenClassification  .I saw this sample code in transformers GIT page. 

from transformers import BertForTokenClassification
tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')
model = BertForTokenClassification.from_pretrained('bert-base-uncased')
input_ids = torch.tensor(tokenizer.encode("Hello, my dog is cute")).unsqueeze(0)  # Batch size 1
labels = torch.tensor([1] * input_ids.size(1)).unsqueeze(0)  # Batch size 1
print(labels)
outputs = model(input_ids, labels=labels)
loss, scores = outputs[:2]



output loss:
tensor(0.5975, grad_fn=<NllLossBackward>)

output scores:

tensor([[[-0.1622,  0.1824],
         [-0.1552, -0.0534],
         [-0.3032, -0.1166],
         [-0.2453, -0.1182],
         [-0.4388, -0.1898],
         [-0.3159, -0.1067]]], grad_fn=<AddBackward0>)

1.When i printed the loss and score i got below values .Now how should i infer this output ? what dose these value represent for  performing NER ? what should i do to get the NER tags for the sentence "Hello, my dog is cute" .


2.i referred few NER codes in GIT using BERT  and they have  humongous line of code written for performing the NER . Is there any simple way to perform NER using bert ?  like how Flair library has very simple method for performing the NER task ? 



Comments:
Why not just use flair? Flair has integrated BERT.

BERT was performing sentence embedding better than Flair(tired all different type of stacked embedding ) but  less when compared to USC . Flair had a functionality that gave NER tagging directly .  i was expecting the same will would be available in BERT but its not available directly .It would be good if BERT gives a direct plug and play functionality for NER task  . 

@ajbot2019 

1. compute prediction list
https://github.com/huggingface/transformers/blob/master/examples/run_ner.py#L242

2. print it with 
https://github.com/huggingface/transformers/blob/master/examples/run_ner.py#L503

shell scripts to train and evaluate for CoNLL 2003 (english) dataset.
https://github.com/dsindex/transformers_examples
this may help you.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Add an LSTM and CNN layer on top of BERT embeddings for sentiment analysis task
Author: johnahug
Pull request: no
Labels: name: wontfix

I am trying to add an LSTM and a convolutional layer on top of my BERT embeddings using the Transformers package in Tensorflow for a sentiment analysis task. Does someone know how I can go about that?

Comments:
Hi @johnahug, I would recommend checking out how the TFBertFor* models work and trying a similar method to add the desired layers via Keras, for example: https://github.com/huggingface/transformers/blob/155c782a2ccd103cf63ad48a2becd7c76a7d2115/transformers/modeling_tf_bert.py#L834

I might be able to help out more at a later time, but I'm not sure exactly when just yet. So it might be worthwhile to go ahead and try it out :)

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


Hi @watkinsm any update on this?

hey @johnahug  have you got what you were looking for? I am also trying to add CNN to my bert embedding but have no idea how to
Fix GPT2 config set to trainable
Author: neonbjb
Pull request: yes

There's currently a bug in the GPT2 model which prevents it from being saved. This is caused by setting the trainable parameter to the GPT2 config, which cannot be packaged later in the save pipeline. Gotta love python...

Here is a simple script which you can use to reproduce this bug (and check the fix):
```
from transformers import (TFGPT2Model)

if __name__ == '__main__':
    _base_model = TFGPT2Model.from_pretrained("gpt2")
    print(base_model._layers[0].trainable)
```

Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/2738?src=pr&el=h1) Report
> Merging [#2738](https://codecov.io/gh/huggingface/transformers/pull/2738?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/9e5b549b4d47678bdc74bc8f650e82cf25bfc245?src=pr&el=desc) will **not change** coverage.
> The diff coverage is `100%`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/2738/graphs/tree.svg?width=650&token=9qOlN6Hb1c&height=150&src=pr)](https://codecov.io/gh/huggingface/transformers/pull/2738?src=pr&el=tree)

```diff
@@           Coverage Diff           @@
##           master    #2738   +/-   ##
=======================================
  Coverage   74.09%   74.09%           
=======================================
  Files          93       93           
  Lines       15249    15249           
=======================================
  Hits        11298    11298           
  Misses       3951     3951
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/2738?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [src/transformers/modeling\_tf\_gpt2.py](https://codecov.io/gh/huggingface/transformers/pull/2738/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ190Zl9ncHQyLnB5) | `94.66% <100%> (ø)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/2738?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/2738?src=pr&el=footer). Last update [9e5b549...5346295](https://codecov.io/gh/huggingface/transformers/pull/2738?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
distilbert_multilingual_cased model for multiple language
Author: divyag11
Pull request: no

By when distilbert_multilingual_cased model is going to be released.because I tried to finetune the 
distilbert_multilingual_cased model , but it said "OSError: file distilbert-base-multilingual-cased not found", which means the above-mentioned model is not included in the list. Or let me know if I am doing any wrong assumption? 

Comments:
Could you try to install a newer version of the library? The `distilbert-base-multilingual-cased` checkpoint is available in recent transformers versions, as you can see in the [source code](https://github.com/huggingface/transformers/blob/master/src/transformers/modeling_distilbert.py#L42).

thanks a lot

hi,
I am trying to finetune distilbert multilingual cased model, but i am getting error while training the model:
error is :
```
ValueError: Error when checking model target: the list of Numpy arrays that you are passing to your model is not the size the model expected. Expected to see 8 array(s), for inputs ['output_1', 'output_2', 'output_3', 'output_4', 'output_5', 'output_6', 'output_7', 'output_8'] but instead got the following list of 1 arrays: [<tf.Tensor 'ExpandDims:0' shape=(None, 1) dtype=int64>]...

while with the same code using distilbert uncased , there is no such error.
Can you please check if there is some problem with distilbert multilingual cased model?

please open the issue

Hello, please open a new issue with information that can help us help you. Your software versions as well as the situation where this error happens being the minimum for us to help you, and a reproducible code example being the most useful.