noamaanMulla-03/transformers-issues huggingface dataset:
question about `add_special_tokens` and embedding layer
Author: hyunwoongko
Pull request: no

# ❓ Questions & Help

when I call add_special_tokens in the tokniezer, new token that is not in the existing vocab is added. At this time, the size is different from the existing embedding layer, what happens inside?

1. Abandon the existing embedding layer and train a new.
2. train only new tokens (expand existing embedding layer)

Comments:
When you add special tokens to the tokenizer, you should resize the model embedding matrix so that it contains the extra embeddings.

You should then do an additional training/fine-tuning on data containing those special tokens so that the model may tune this new column in the embedding matrix.

thanks !
How to create a QA model where the answer can be from the question text as well?
Author: nrjvarshney
Pull request: no

# ❓ Questions & Help

<!-- The GitHub issue tracker is primarly intended for bugs, feature requests,
     new models and benchmarks, and migration questions. For all other questions,
     we direct you to the Hugging Face forum: https://discuss.huggingface.co/ .
     You can also try Stack Overflow (SO) where a whole community of PyTorch and
     Tensorflow enthusiast can help you out. In this case, make sure to tag your
     question with the right deep learning framework as well as the
     huggingface-transformers tag: 
     https://stackoverflow.com/questions/tagged/huggingface-transformers 
     -->

## Details
<!-- Description of your issue -->
SQuAD QA dataset has questions where the answer is a span in the context, 
How do I create a QA system where the answer string can be from the context or question text?



<!-- You should first ask your question on the forum or SO, and only if
     you didn't get an answer ask it here on GitHub. -->


Comments:
Normally there's no difference. You just provide `[CLS] question [SEP] answer [SEP]` examples to the model, and the `start_positions` and `end_positions` can be indexes of the question. 

Yes,
Can you point me to such an implementation?
huggingface Squad Question answering code handles a lot of edge cases like start token should be before the end token, find n best predictions etc. 
@NielsRogge , Is there a simpler implementation for the question answering task?

I found one in the documentation but that doesn't handle the predictions, sanity checks, n_best predictions etc.
How to enable sampling when using unigram tokenizers?
Author: dennisylyung
Pull request: no
Labels: name: wontfix

# ❓ Questions & Help

## Details
It is mentioned on [Tokenizer summary][1] that when using unigram tokenizers, "you could sample one of the tokenization according to their probabilities". It however did not detail how to do so.

Looking at the source-code, all of `AlbertTokenizer`, `T5Tokenizer` and `XLNetTokenizer` have an argument "sample" in their `_tokenize()` method, which in turns call the encoding method from sentencepiece with sampling enabled. 
```
    def _tokenize(self, text, sample=False):
        """ Tokenize a string. """
        text = self.preprocess_text(text)

        if not sample:
            pieces = self.sp_model.EncodeAsPieces(text)
        else:
            pieces = self.sp_model.SampleEncodeAsPieces(text, 64, 0.1)
        ...
```
But when I check the usages of _tokenize, it seems to be only called without a "sample" argument, in `PreTrainedTokenizer`. 
```
        def split_on_tokens(tok_list, text):
            if not text.strip():
                return []
            if not tok_list:
                return self._tokenize(text)
```

I tried calling the `encode()` method of the tokenizer class with `sample=True`, but the keyword was not recognized. 
```
albert_tokenizer = AlbertTokenizer('m.model')
albert_tokenizer.encode(msg, sample=True)

>> Keyword arguments {'sample': True} not recognized.
```

How should I enable sampling when using unigram tokenizers?


  [1]: https://huggingface.co/transformers/tokenizer_summary.html#unigram
**A link to original question on the forum/Stack Overflow**:
[https://stackoverflow.com/questions/63436152/how-to-enable-sampling-when-using-unigram-tokenizers-in-huggingface](url)

Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
tokenizers/tokenizers.cpython-36m-darwin.so, 2): Symbol not found: ____chkstk_darwin 
Author: orenpapers
Pull request: no
Labels: name: wontfix

Hello
I am using jupyter notebook and installed tranformers with:
`!{sys.executable} -m pip install transformers 
`
I have MacOS 10.13
When trying to import the tranformers:
`from transformers import BertTokenizer, BertModel
`
I get the error:
```
ImportError: dlopen(/Users/user1/anaconda3/lib/python3.6/site-packages/tokenizers/tokenizers.cpython-36m-darwin.so, 2): Symbol not found: ____chkstk_darwin
  Referenced from: /Users/user1/anaconda3/lib/python3.6/site-packages/tokenizers/tokenizers.cpython-36m-darwin.so (which was built for Mac OS X 10.15)
  Expected in: /usr/lib/libSystem.B.dylib
 in /Users/user1/anaconda3/lib/python3.6/site-packages/tokenizers/tokenizers.cpython-36m-darwin.so
```

Any idea how to fix it?

Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Model Clipping reduce the size of the model.
Author: snaik2016
Pull request: no
Labels: name: wontfix

# ❓ Questions & Help

Is it possible to just reduce the size of a large model to by removing weights and preserving others.
For example we can apply this concept to different dimensions
Depth: Layer dropping this is well known an have been shown to work with huggingface repo.
Width: For scenarios where we need small size of the model, is it possible to just drop all the weights beyond all the seq length L.
For GPT2 kind of model with only left to right flow it shouldn't affect the model perf.
How to implement this without disturbing other weights?

Thanks in advance for help!

<!-- The GitHub issue tracker is primarly intended for bugs, feature requests,
     new models and benchmarks, and migration questions. For all other questions,
     we direct you to the Hugging Face forum: https://discuss.huggingface.co/ .
     You can also try Stack Overflow (SO) where a whole community of PyTorch and
     Tensorflow enthusiast can help you out. In this case, make sure to tag your
     question with the right deep learning framework as well as the
     huggingface-transformers tag: 
     https://stackoverflow.com/questions/tagged/huggingface-transformers 
     -->

## Details
<!-- Description of your issue -->

<!-- You should first ask your question on the forum or SO, and only if
     you didn't get an answer ask it here on GitHub. -->
**A link to original question on the forum/Stack Overflow**:

Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Longformer Memory Consumption query
Author: PrudhviRaj12
Pull request: no
Labels: name: wontfix

Hello,
Apologies if I am misunderstanding it, but if I use roberta with a max sequence length of 256 and I can run, for example, say, a batch size of 64 on one gpu for a task, with the longformer, can I use the same batch size with a window length of 256 with the max sequence length being 4096?

Comments:
A question that might be of interest to @patrickvonplaten :)

Hey @PrudhviRaj12, 
my best answer would be to try it out :-) If can definitely work if you have enough GPU RAM. 
My best guess would be that in the scenario described by you above the Longformer version would require ca. `num_chunks` * `required_gpu_ram_for_roberta` and `num_chunks` in your case would be 4096 / 256 = 16. So you would need a lot of RAM to run `batch_size=64` and `max_length=4096` with Longformer, most likely not enough for one GPU (even if fp16).

I would also suggest adding gradient_checkpointing=True when you load your model with from_pretrained. This recent addition to the HF code base allowed me to go from using BERT with a max sequence length of 128-256 before running out of memory to now being able to use Longformer with a max seq length of up to 4096 on the same GPU setup!

This thread helped me and may also help you:
https://github.com/allenai/longformer/issues/80

Thanks @patrickvonplaten  - I misunderstood the paper then. @HugToDebug thanks for your suggestion - I tried that but I am getting this warning 

```
None of the inputs have requires_grad=True. Gradients will be None
warnings.warn("None of the inputs have requires_grad=True. Gradients will be None"
```

when calling model.forward(sequence inputs, attention masks) with any model (be it longformer or bert or roberta) and the performance of the model is completely off of the same batch experimental setting with and without gradient checkpointing. Probably I am doing something wrong, I'll check that thread. 

I am only training the last N layers of the bert/roberta for my task, and I am setting requires grad = False for all the other layers and I am getting that warning. When I remove that condition of setting requires grad = False for some layers and leaving them true for all, I am not getting that warning. Any idea how to get around that issue?

Update: I was able to get rid of that warning by making one of the embedding weight matrices trainable (in my case - Roberta, token type embedding). It was only adding 768 more trainable parameters, but I am getting OOM. I had to cut down the batch size 4x to get it running on one gpu without OOM. Not sure why adding just 768 trainable params had that of an impact.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Pretokenized text handling mistake in  tokenization_utils.py file
Author: syuoni
Pull request: no
Labels: name: wontfix

Hi, 

I'm wondering if this is a mistake that in the `tokenization_utils.py` file, the pretokenized text will be further tokenized, while the raw text would not. The code is as follows: 

```
if is_pretokenized:
    tokens = list(itertools.chain(*(self.tokenize(t, is_pretokenized=True, **kwargs) for t in text)))
         return self.convert_tokens_to_ids(tokens)
    else:
        return self.convert_tokens_to_ids(text)
```


Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Create my own language model
Author: dhimasyoga16
Pull request: no
Labels: name: wontfix

I'm new in NLP and i want to give BERT a try. I have a wikipedia corpus (in Indonesian language of course, and in .txt format) and want to train it with bert multilingual cased.
For further use, i expect that BERT can "adapt" well with indonesian language and can do specific task which is a text similarity task, or if possible do automated scoring based on the 2 texts given.

Can i use the [run_language_modeling.py](https://github.com/huggingface/transformers/blob/master/examples/language-modeling/run_language_modeling.py) to fine-tune and create my own language model?
If it's possible, then what are the exact steps to achieve this?

Thankyou in advance.



Comments:
For open-ended questions like this you should try https://discuss.huggingface.co

Did you read https://huggingface.co/blog/how-to-train?

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
New Feature: Best-First Beam Search
Author: JetRunner
Pull request: no
Labels: name: wontfix

# 🚀 Feature request

<!-- A clear and concise description of the feature proposal.
     Please provide a link to the paper and code in case they exist. -->

Best-First Beam Search is a faster beam search algorithm for decoding.
https://arxiv.org/abs/2007.03909 (By Clara Meister, Tim Vieira, Ryan Cotterell)

## Motivation

This technique can be essential for production-purpose inference and of great interest to our users. It would be a good idea to integrate Best-First Beam Search to Hugging Face transformers (for GPT, BART, T5, etc.).

Comments:
The first step would be refactoring our current implementation with the priority queue. I am on it.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
tensorflow bert model can‘t  return all hidden_states 
Author: SunYanCN
Pull request: no
Labels: name: wontfix

```
import tensorflow as tf
import tensorflow_datasets
from transformers import *
from tensorflow.keras import layers

configuration = BertConfig.from_pretrained('bert-base-cased', output_hidden_states=True)
# Load dataset, tokenizer, model from pretrained model/vocabulary
tokenizer = BertTokenizer.from_pretrained('bert-base-cased', config=configuration)
# BERT encoder
encoder = TFBertModel.from_pretrained('bert-base-cased', config=configuration)
# Model
input_ids = layers.Input(shape=(100,), dtype=tf.int32)
token_type_ids = layers.Input(shape=(100,), dtype=tf.int32)
attention_mask = layers.Input(shape=(100,), dtype=tf.int32)
outputs = encoder(
    input_ids, token_type_ids=token_type_ids, attention_mask=attention_mask
)
print(outputs)

_, _, hidden_states = outputs[0], outputs[1], outputs[2]
```

output:

```
If your task is similar to the task the model of the ckeckpoint was trained on, you can already use TFBertModel for predic
(<tf.Tensor 'tf_bert_model_2/Identity:0' shape=(None, 100, 768) dtype=float32>, <tf.Tensor 'tf_bert_model_2/Identity_1:0' shape=(None, 768) dtype=float32>)


---------------------------------------------------------------------------
IndexError                                Traceback (most recent call last)
<ipython-input-7-63944e137dcd> in <module>()
     17 )
     18 print(outputs)
---> 19 _, _, hidden_states = outputs[0], outputs[1], outputs[2]

IndexError: tuple index out of range
```

You can check it:
[colab code](https://colab.research.google.com/drive/1zZo2SwBuGoCHsbvL3_fVR5rygWiuR16-?usp=sharing)

Comments:
I'm encountering the exact same error - and it also happens when trying to output the attention keys. 
I would really like to know where the problem comes from as well! 

@sshleifer 

@julien-c 

transformers==2.7.0 solved the issue in my case

> transformers==2.7.0 solved the issue in my case

Thanks a lot. It solved my case too

Thank you so much @VietHoang1710, worked for me - The same problem with tensorflow roberta model, lost a lot of time to this one!

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
ValueError: Unable to set proper padding strategy as the tokenizer does not have a padding token. 
Author: banunitte
Pull request: no

# ❓ Questions & Help

<!-- The GitHub issue tracker is primarly intended for bugs, feature requests,
     new models and benchmarks, and migration questions. For all other questions,
     we direct you to Stack Overflow (SO) where a whole community of PyTorch and
     Tensorflow enthusiast can help you out. Make sure to tag your question with the
     right deep learning framework as well as the huggingface-transformers tag: 
     https://stackoverflow.com/questions/tagged/huggingface-transformers 
     
     If your question wasn't answered after a period of time on Stack Overflow, you
     can always open a question on GitHub. You should then link to the SO question 
     that you posted.
     -->

## Details
Using pad_token, but it is not set yet. 

tokenizer = GPT2Tokenizer.from_pretrained('gpt2')

tokens = tokenizer.batch_encode_plus(
    ["This is a sample", "This is another longer sample text"], 
    pad_to_max_length=True ,max_length=10 ,return_attention_mask = True# First sentence will have some PADDED tokens to match second sequence length
)

for i in range(2):
    print("Tokens (int)      : {}".format(tokens['input_ids'][i]))
    print("Tokens (str)      : {}".format([tokenizer.convert_ids_to_tokens(s) for s in tokens['input_ids'][i]]))
    print("Tokens (attn_mask): {}".format(tokens['attention_mask'][i]))
    print()


Comments:
![pad_token_not_Set](https://user-images.githubusercontent.com/6847024/79693840-9f6c6400-828a-11ea-9760-ef3f8a12c6ae.PNG)


tokenizer.pad_token = 0

You have to set the pad_token_id yourself as it's stated in the error message ;-). I would recommend using the `eos_token_id` as the `pad_token_id` for GPT2:
```python 
tokenizer = GPT2Tokenizer.from_pretrained('gpt2')
tokenizer.pad_token = tokenizer.eos_token
```
as it's written in the error message ;-)

I hit the same issue after using [add_special_tokens](https://huggingface.co/transformers/main_classes/tokenizer.html#transformers.PreTrainedTokenizer.add_special_tokens) with `{"pad_token": "PAD"}` dictionary.

I understand based on the error and the documentation it should not raise the error, right? @patrickvonplaten should the issue be reopened?

For completeness: 
- patrickvonplaten tip `tokenizer.pad_token = tokenizer.eos_token` solved it
- the whole error message for me was

```
ValueError: Unable to set proper padding strategy as the tokenizer does not have a padding token. In this case please set the `pad_token` `(tokenizer.pad_token = tokenizer.eos_token e.g.)` 
or add a new pad token via the function add_special_tokens if you want to use a padding strategy
```

You didn't set pad_token. You can set it like this:
```python
tokenizer.pad_token = "[PAD]"
```
Can not convert the pytorch pretrained bert model to onnx model 
Author: KellyZhang2020
Pull request: no
Labels: name: wontfix

When I convert the pytorch pretrained bert model to onnx model  as follows:

```
import os
import torch
from pytorch_pretrained_bert import BertTokenizer, BertModel

model = BertModel.from_pretrained('bert-base-uncased')

input_1 = torch.LongTensor(1, 14)
input_2 = torch.LongTensor(1, 14)
input_3 = torch.LongTensor(1, 14)
base_path = os.path.dirname(__file__)
onnx_path = os.path.join(base_path, 'bert.onnx')
torch.onnx.export(model, (input_1, input_2, input_3), onnx_path)
```
**raise error**: 
`IndexError: index out of range in self`
     
- `transformers` version: 3.0.2
- Platform: pycharm
- Python version: 3.8
- PyTorch version (GPU?): 1.6.0 no GPU
- onnx version: 1.7.0
- pytorch-pretrained-bert version: 0.6.2

Thank you for the help :)




Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
EncoderDecoderModel loss function
Author: AI678
Pull request: no

# ❓ Questions & Help

<!-- The GitHub issue tracker is primarly intended for bugs, feature requests,
     new models and benchmarks, and migration questions. For all other questions,
     we direct you to the Hugging Face forum: https://discuss.huggingface.co/ .
     You can also try Stack Overflow (SO) where a whole community of PyTorch and
     Tensorflow enthusiast can help you out. In this case, make sure to tag your
     question with the right deep learning framework as well as the
     huggingface-transformers tag: 
     https://stackoverflow.com/questions/tagged/huggingface-transformers 
     -->

## Details
<!-- Description of your issue -->
Hey , I want to ask the following questions.
How is the loss calculated in DecoderEncoderModel. What is the mathematic formula of the loss function ? 
I just wrote the code like this 

outputs = model(input_ids=src, attention_mask=mask, decoder_input_ids=dst, labels=dst, return_dict=True)
loss, logits = outputs.loss, outputs.logits

<!-- You should first ask your question on the forum or SO, and only if
     you didn't get an answer ask it here on GitHub. -->
**A link to original question on the forum/Stack Overflow**:

Comments:
Hello, this depends on the decoder you use to initialize the encoder-decoder model. What decoder do you use?

I use 'bert-base-uncased'. just like this 
model = EncoderDecoderModel.from_encoder_decoder_pretrained('bert-base-uncased', 'bert-base-uncased')

I'm not sure this is the recommended way to load the models as it gives the following result:

```
Some weights of BertLMHeadModel were not initialized from the model checkpoint at bert-base-uncased and are newly initialized: ['bert.encoder.layer.0.crossattention.self.query.weight', 'bert.encoder.layer.0.crossattention.self.query.bias', [...]
```
with pretty much all model weights.

Will ping @patrickvonplaten for advice.

Hey @AI678, 

1) The model should be initialized just as you did with

```python
model = EncoderDecoderModel.from_encoder_decoder_pretrained('bert-base-uncased', 'bert-base-uncased')
```

It's normal that `None` of the cross-attention layers are initialized because BERT does not have any and they have to be fine-tuned down the road. 

2) To Train a Bert2Bert, you are also correct in doing: 

```python
outputs = model(input_ids=src, attention_mask=mask, decoder_input_ids=dst, labels=dst, return_dict=True)
loss, logits = outputs.loss, outputs.logits
```

because BERT automatically shifts the labels for you, see: https://github.com/huggingface/transformers/blob/901e9b8eda2fe88af717f960ddc05cac1803679b/src/transformers/modeling_bert.py#L1060

Also I'll publish a more in-detail notebook about "Leveraging Encoder-Decoder models" soon. This model card could also be helpful: https://huggingface.co/patrickvonplaten/bert2bert-cnn_dailymail-fp16#bert2bert-summarization-with-%F0%9F%A4%97-encoderdecoder-framework


thank you very much