Question Answering Modeling through Hugging Face Models

# ❓ Questions & Help

<!-- The GitHub issue tracker is primarly intended for bugs, feature requests,
     new models and benchmarks, and migration questions. For all other questions,
     we direct you to Stack Overflow (SO) where a whole community of PyTorch and
     Tensorflow enthusiast can help you out. Make sure to tag your question with the
     right deep learning framework as well as the huggingface-transformers tag:
     https://stackoverflow.com/questions/tagged/huggingface-transformers

     If your question wasn't answered after a period of time on Stack Overflow, you
     can always open a question on GitHub. You should then link to the SO question
     that you posted.
     -->

## Details
<!-- Description of your issue -->
Hello everyone, I have large number of documents and I need to extract specific information through Hugging Face Question Answering Model. First issue I faced was the document size was very large so it gave me token error and afterwards, I divided the data into small paragraphs, then I applied the given model. But this time, answer was not accurate. So, I just want to know, is there any alternative method or model to do this.

<!-- You should first ask your question on SO, and only if
     you didn't get an answer ask it here on GitHub. -->
**A link to original question on Stack Overflow**:


Create model card for T5-base fine-tuned for Sentiment Span Extraction

model card

Related to pretrained model cards


BertModel Inputs

# ❓ Questions & Help

Hi,

I have been using BertModel for Question and Answering . In examples - I see there are no start_position , end_position being provided to the Model .

How is the Model able to train in this case using Input_ids , Mask and Attention Head ?

Might be a naive question , but I have dig into the source codes and referred to runsquad.py - could not gain clarity.

Can anybody suggest?

Thanks

<!-- The GitHub issue tracker is primarly intended for bugs, feature requests,
     new models and benchmarks, and migration questions. For all other questions,
     we direct you to Stack Overflow (SO) where a whole community of PyTorch and
     Tensorflow enthusiast can help you out. Make sure to tag your question with the
     right deep learning framework as well as the huggingface-transformers tag:
     https://stackoverflow.com/questions/tagged/huggingface-transformers

     If your question wasn't answered after a period of time on Stack Overflow, you
     can always open a question on GitHub. You should then link to the SO question
     that you posted.
     -->

## Details
<!-- Description of your issue -->

<!-- You should first ask your question on SO, and only if
     you didn't get an answer ask it here on GitHub. -->
**A link to original question on Stack Overflow**:


Bart model for textinfilling

# ❓ Questions & Help

## Details
<!-- Description of your issue -->
@sshleifer
I was wandering why you used a registered_buffer to define the ``final_logits_bias`` in the BartForConditionalGeneration. In my understanding a registered_buffer would have not gradient so if I fine-tune the a Bart model for generation these biases would not be updated (remain to 0). Should we use  register_parameter or remove the bias completely or there is something I haven't understand ?

<!-- You should first ask your question on SO, and only if
     you didn't get an answer ask it here on GitHub. -->
**A link to original question on Stack Overflow**:


TFTrainer: Checkpoints not getting saved in `output_dir` but in {cwd}/checkpoint

I am using TFTrainer for the SQuAD task.
Checkpoints are being created in cwd/checkpoint insted of output_dir.

**Potential Cause:**
https://github.com/huggingface/transformers/blob/9ca485734aea269961d63a040ff194365d151fd1/src/transformers/trainer_tf.py#L156
Instead of PREFIX_CHECKPOINT_DIR we need to have
```python
os.path.join(self.args.output_dir, PREFIX_CHECKPOINT_DIR)
```


When I use TFBertEncoder in my laptop, I get an error.I can not build a model. Here is a simple examples.

# 🐛 Bug

## Information

Model I am using TFBertEncoder:

Language I am using the model on English:

The problem arises when using:
* [ ] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:

1. When I use, TFBertEncoder, I get an error.

Here is my code.
```py
import tensorflow as tf
import numpy as np
from transformers.modeling_tf_bert import BertConfig, TFBertEncoder

print(tf.__name__, tf.__version__)
input_a = tf.keras.layers.Input(shape=(91, 128))

config = BertConfig()
config.hidden_size = 128
config.num_attention_heads = 4

# config.output_attentions = False
# config.output_hidden_states = False

head_mask = [None for _ in range(config.num_hidden_layers)]
encoder_output = TFBertEncoder(config=config)([input_a, None, head_mask])[0]
print(encoder_output.shape)

test_out = tf.keras.layers.Dense(128)(encoder_output)
print(test_out.shape)
```
## Expected behavior
Here is the error:

```
(None, 91, 128)
2020-06-03 11:18:10.160647: W tensorflow/core/common_runtime/base_collective_executor.cc:217] BaseCollectiveExecutor::StartAbort Failed precondition: Error while reading resource variable _AnonymousVar189 from Container: localhost. This could mean that the variable was uninitialized. Not found: Resource localhost/_AnonymousVar189/class tensorflow::Var does not exist.
	 [[{{node output_23/dense/BiasAdd/ReadVariableOp}}]]
Traceback (most recent call last):
  File "D:/python/tx/TEST.py", line 16, in <module>
    a = tf.keras.layers.Dense(128)(encoder_output)
  File "D:\Anaconda3\lib\site-packages\tensorflow_core\python\keras\engine\base_layer.py", line 720, in __call__
    base_layer_utils.create_keras_history(inputs)
  File "D:\Anaconda3\lib\site-packages\tensorflow_core\python\keras\engine\base_layer_utils.py", line 187, in create_keras_history
    _, created_layers = _create_keras_history_helper(tensors, set(), [])
  File "D:\Anaconda3\lib\site-packages\tensorflow_core\python\keras\engine\base_layer_utils.py", line 249, in _create_keras_history_helper
    layer_inputs, processed_ops, created_layers)
  File "D:\Anaconda3\lib\site-packages\tensorflow_core\python\keras\engine\base_layer_utils.py", line 249, in _create_keras_history_helper
    layer_inputs, processed_ops, created_layers)
  File "D:\Anaconda3\lib\site-packages\tensorflow_core\python\keras\engine\base_layer_utils.py", line 249, in _create_keras_history_helper
    layer_inputs, processed_ops, created_layers)
  [Previous line repeated 5 more times]
  File "D:\Anaconda3\lib\site-packages\tensorflow_core\python\keras\engine\base_layer_utils.py", line 247, in _create_keras_history_helper
    constants[i] = backend.function([], op_input)([])
  File "D:\Anaconda3\lib\site-packages\tensorflow_core\python\keras\backend.py", line 3727, in __call__
    outputs = self._graph_fn(*converted_inputs)
  File "D:\Anaconda3\lib\site-packages\tensorflow_core\python\eager\function.py", line 1551, in __call__
    return self._call_impl(args, kwargs)
  File "D:\Anaconda3\lib\site-packages\tensorflow_core\python\eager\function.py", line 1591, in _call_impl
    return self._call_flat(args, self.captured_inputs, cancellation_manager)
  File "D:\Anaconda3\lib\site-packages\tensorflow_core\python\eager\function.py", line 1692, in _call_flat
    ctx, args, cancellation_manager=cancellation_manager))
  File "D:\Anaconda3\lib\site-packages\tensorflow_core\python\eager\function.py", line 545, in call
    ctx=ctx)
  File "D:\Anaconda3\lib\site-packages\tensorflow_core\python\eager\execute.py", line 67, in quick_execute
    six.raise_from(core._status_to_exception(e.code, message), None)
  File "<string>", line 3, in raise_from
tensorflow.python.framework.errors_impl.FailedPreconditionError:  Error while reading resource variable _AnonymousVar189 from Container: localhost. This could mean that the variable was uninitialized. Not found: Resource localhost/_AnonymousVar189/class tensorflow::Var does not exist.
	 [[node output_23/dense/BiasAdd/ReadVariableOp (defined at /python/tx/TEST.py:16) ]] [Op:__inference_keras_scratch_graph_5205]

Function call stack:
keras_scratch_graph
```

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 2.3.0 (in conda list)
- Platform:
- Python version:3.7
- PyTorch version (GPU?):
- Tensorflow version (GPU?):TF2.1.0(GPU)
- Using GPU in script?:
- Using distributed or parallel set-up in script?:No


Adding notebooks for Fine Tuning [Community Notebook]

Hi @patrickvonplaten, Adding 3 documented notebooks to fine tune transformers to downstream NLP tasks with PyTorch:

- Multi-class classification: Using DistilBert
- Multi-label classification: Using Bert
- Summarization: Using T5 - **Model Tracking with WandB**

These notebooks are pulled from the git repo: https://github.com/abhimishra91/transformers-tutorials


[DOT NOT MERGE] Tokenizers Shape Polymorphism - Introduce pad_to_next_multiple_of parameters

Needs new release of tokenizers (cc @n1t0)


bert-small-cord19 model cards

Related to pretrained model cards

Adds model cards for the bert-small-cord19 series of models.


[Feature request] Support batched conditional generation from GPT-2

# 🚀 Feature request

Support batched conditional generation from GPT-2

## Motivation

Currently the [method](https://github.com/huggingface/transformers/blob/9ca485734aea269961d63a040ff194365d151fd1/src/transformers/modeling_utils.py#L802) to generate text from GPT-2 conditioned on an input sequence only supports either 1) a single input at a time, or 2) a batch of inputs where the conditioning input sequence is the same length. It would be great (for efficiency) if this method could be updated to support a batch with conditional inputs of varying length, done by ignoring padding in the input_ids.

## Your contribution

Unlikely to have time to code this, but will submit a PR if I do.


Possible fix to make AMP work with DDP in the trainer

PyTorch

Anything PyTorch

closes https://github.com/huggingface/transformers/issues/4657

Using multiple GPUs in PyTorch (with DistrubutedDataParallel) speeds up performance drastically. To get even more speed out of it, the example scripts often - if not always - allow the use of apex for automatic mixed precision. This is great. However, some issues can arise, particularly the infamous "illegal memory access" error which seems to have an easy, one-line solution.

Currently, we assume that by using things like `.to(args.device)` we solve any and all issues of where our data or model should go. However, the author of AMP @mcarilli seems to [suggest](https://github.com/NVIDIA/apex/issues/319#issuecomment-503372924) that it is recommended to always set the current process' default device, too, to ensure no further issues. This suggestion also [helped](https://github.com/huggingface/transformers/issues/4657#issuecomment-637703146) the aforementioned issue. Therefore it seems a good idea to also implement this. In fact, some examples such as Hans already do this.

https://github.com/huggingface/transformers/blob/b231a413f5d58592bb4d98304c3d3b668c5d4a42/examples/adversarial/test_hans.py#L518

To avoid DRY issues, I would suspect that the trainer_args file is the best place to do this _only once_ but other suggestions are welcome (this is different from what I suggested in the linked issue, though I think `trainer_args` is the better place). I am not sure which examples do not use trainer_args, but those would need to be checked and updated as well. If anyone can give a quick rundown of which examples do NOT use the new trainer, I can have a look quickly. Otherwise I'll have to go over the examples another time.

**Side note**: I am not sure how and if this works with TPUs so to be sure that this only involves CUDA devices, I first check whether the device is a CUDA device.


Albert pretraining loss not decreasing

I am training Albert from scratch using run_language_modeling.py.
Doing training on 8 V100.p3dn8x.
Launching with this parameters.

```
python transformers/examples/language-modeling/test.py   --train_data_file x.txt  --output_dir albert_model --model_type albert --mlm --config_name test --tokenizer_name test --do_train --line_by_line --learning_rate 0.00088 --num_train_epochs 3 --save_total_limit 50 --save_steps 5000 --per_gpu_train_batch_size 150 --seed 42 --overwrite_output_dir --max_steps 200000 --fp16
```
![image](https://user-images.githubusercontent.com/18630864/83558269-65041100-a530-11ea-965b-f24a76550a28.png)

loss is not decreasing. The above plot contains training with and without warmup steps training curve. loss is stuck on `7.27` in both case.
Also, weird thing is while I am setting the number of epochs 3 but in training its showing 9
```
was: ModuleNotFoundError("No module named 'amp_C'",)
06/02/2020 14:45:14 - INFO - transformers.trainer -   ***** Running training *****
06/02/2020 14:45:14 - INFO - transformers.trainer -     Num examples = 28236463
06/02/2020 14:45:14 - INFO - transformers.trainer -     Num Epochs = 9
06/02/2020 14:45:14 - INFO - transformers.trainer -     Instantaneous batch size per device = 150
06/02/2020 14:45:14 - INFO - transformers.trainer -     Total train batch size (w. parallel, distributed & accumulation) = 1200
06/02/2020 14:45:14 - INFO - transformers.trainer -     Gradient Accumulation steps = 1
06/02/2020 14:45:14 - INFO - transformers.trainer -     Total optimization steps = 200000
```
Can anyone suggest what could be reasons for such behavior?


TFRobertaModelIntegrationTest requires tf


Save & load sparse models from the models database

# 🚀 Feature request

<!-- A clear and concise description of the feature proposal.
     Please provide a link to the paper and code in case they exist. -->

Handle saving and loading pipeline for sparse models such as PruneBERT.

## Motivation

<!-- Please outline the motivation for the proposal. Is your feature request
     related to a problem? e.g., I'm always frustrated when [...]. If this is related
     to another GitHub issue, please link here too. -->

New sparse models will be extremely useful if we can use them by downloading the compressed. version with the `.from_pretrained` functions

## Your contribution

<!-- Is there any way that you could help, e.g. by submitting a PR?
     Make sure to read the CONTRIBUTING.MD readme:
     https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md -->

An example is already part of the current repo so I can try to create a PR later: https://github.com/huggingface/transformers/blob/master/examples/movement-pruning/Saving_PruneBERT.ipynb


Fix CI after killing archive maps


never_split on slow tokenizers should not split

I'm actually not sure if it's the right behavior, but when using `do_basic_tokenization` on `BertTokenizer` the parameter `never_split` is not used to determine if a token should be sent to wordpiece tokenizer.

This PR checks, for each tokens returned by `basic_tokenizer`, if the token is not in the `never_split: set` before sending to wordpiece. If the token is found in `never_split` then it is added as-it in the returned list of tokens.

Updated `never_split: List` -> `never_split: Set` as we're always testing for membership in the set and not for a specific index in the collection. [Set are 10x faster for membership operations than list](https://stackoverflow.com/a/17945009)

**Before:**

```python
tokenizer = BertTokenizer.from_pretrained(
    "bert-base-cased",
    use_fast=False,
    never_split=['lol'],
    do_basic_tokenize=True
)
tokenizer.tokenize("lol")
Out[4]: ['lo', '##l']
```

**After**

```python
tokenizer = BertTokenizer.from_pretrained(
    "bert-base-cased",
    use_fast=False,
    never_split=['lol'],
    do_basic_tokenize=True
)
tokenizer.tokenize("lol")
Out[5]: ['lol']
```

Related to #3518


Unify label args

Following up from #4711, this is a proposal to deprecate any argument that's not `labels` (like `madked_lm_labels`, `lm_labels`, etc.) to `labels`.

I've only done one model for now to get feedback on the design, once we have something you like, I can do them all (or have separate PRs if you think that's best).


Faster bert basic tokenizer

In this PR I tried 2 things:

* First I change some comparisons in the form of `a == sth or a == otherthing` for `a in {sth, otherthing}`. It is faster and more readable.

* I noticed that some methods could actually be functions because they did not use anything from the class.

I am a bit new to this, so I will accept any feedback.


[Reformer] Improved memory if input is shorter than chunk length

This PR improves memory and speed of Reformer for language generatinon.

Reformer is based on chunked self attention. This means that for an input length which is not a multiple of the chunk length, the input has to be padded to be a multiple of the chunk length. This is not the case though when the input length is less than the chunk length (happens in language generation). In this case, normal self attention should be applied to save memory.

Code is updated for both LSH and Local self attention and a test is added.


About `do_basic_tokenize` behavior in BertTokenizer

# ❓ Questions & Help

In the [docs](https://huggingface.co/transformers/model_doc/bert.html?highlight=basic%20tokenization#berttokenizer) it just says that it performs a basic tokenization before the word piecer. But what is actually a "basic tokenization"? I would really appreciate a little more of information on that.

Diving into the code I found out that the basic tokenization removes control characters from the text. I did not expect that behavior from what I read on the docs. That gave us problems because some characters like `` weren't tokenizing and we didn't know why.

More generally I think that transformers has a problem with the docs. Most of the times that something does not work for me, I don't bother to look at them and I directly dive into the source code in order to understand what's happening. I am used to scikit-learn and maybe I am a bit bisaed, but I really think that these kind of things can be a barrier for new people wanting to use transformers.

If there is something I can do to help, I am happy to send a PR.


Replace pad_token with -100 for LM loss calculation

The docs for both GPT and [GPT2](https://huggingface.co/transformers/model_doc/gpt2.html#gpt2lmheadmodel) specify that labels that are not -100 will be used for the calculation of the loss. So, the padding for the labels should be `-100`, not `tokenizer.pad_token_id`.


Override get_vocab for fast tokenizer.


Can I save a word embedding from BERT and used again later for computational purpose(as BERT takes much more time). Just like in Glove. If yes then How? and is this good idea to do so?

# ❓ Questions & Help

<!-- The GitHub issue tracker is primarly intended for bugs, feature requests,
     new models and benchmarks, and migration questions. For all other questions,
     we direct you to Stack Overflow (SO) where a whole community of PyTorch and
     Tensorflow enthusiast can help you out. Make sure to tag your question with the
     right deep learning framework as well as the huggingface-transformers tag:
     https://stackoverflow.com/questions/tagged/huggingface-transformers

     If your question wasn't answered after a period of time on Stack Overflow, you
     can always open a question on GitHub. You should then link to the SO question
     that you posted.
     -->

## Details
<!-- Description of your issue -->

<!-- You should first ask your question on SO, and only if
     you didn't get an answer ask it here on GitHub. -->
**A link to original question on Stack Overflow**:


how to make a multi-task deep neural network baseline using huggingface transformers?

I was trying to build a [multi-task deep neural network][1] using [xlm roberta large model][2] for a multilingual classification problem. my training dataset contains 4 columns :

1. ID

2. comment_text (according to id number,each users english comment is stored in this column. example comment : "you are a loser")

3. toxic (this column contains 1/0,0 means not toxic,1 means toxic)

4. personal_attack(this column also contains 0/1,,0 means the comment is not a personal attack type comment and 1 means opposite)

here is my models code :

    def build_model(transformer, max_len=512):
        input_word_ids = Input(shape=(max_len,), dtype=tf.int32, name="input_word_ids")
        sequence_output = transformer(input_word_ids)[0]
        cls_token = sequence_output[:, 0, :]
        out = Dense(1, activation='sigmoid',name = 'y_train')(cls_token)
        out1 = Dense(1, activation='sigmoid',name = 'y_aux')(cls_token)
        model = Model(inputs=input_word_ids, outputs=[out, out1])
        model.compile(Adam(lr=1e-5), loss='binary_crossentropy', metrics=['accuracy'])

        return model

here is the code for train and test dataset :

    train_dataset = (
    tf.data.Dataset
    .from_tensor_slices((x_train,

    { 'y_train':train.toxic.values,
    'y_aux':train.identity_attack.values}

    ))
    .repeat()
    .shuffle(2048)
    .batch(BATCH_SIZE)
    .prefetch(AUTO)
    )

    test_dataset = (
        tf.data.Dataset
        .from_tensor_slices(x_test)
        .batch(BATCH_SIZE)
    )

then for training model i used this code :

    EPOCHS = 3
    n_steps = x_train.shape[0] // BATCH_SIZE
    train_history = model.fit(
        train_dataset,
        steps_per_epoch=n_steps,
        epochs=EPOCHS
    )

i don't wish to perform validation so just train_dataset was given for model.fit()

after 3 epoch i get performance like this :

    Epoch 3/3
    1658/1658 [==============================] - 887s 535ms/step - loss: 0.0591 - y_train_loss: 0.0175 - y_aux_loss: 0.0416 - y_train_accuracy: 0.9940 - y_aux_accuracy: 0.9821

now in my test set i have 1 columns :

1. comments( this column contains comments of non english language (remember in train set we only had English comments and here in test set all comments are non english)

so i expect my model to predict on these test set whether the given test set comment is toxic or not?
as you can see from 3rd epoch result that i am calculating y_train_accuracy: 0.9940 - y_aux_accuracy: 0.9821

now i want my model to predict y_test or toxic/not toxic only

for that i tried :

    sub['toxic'] = model.predict(test_dataset, verbose=1)

sub is a dataframe that contains all the id of test set and using **test_dataset** i was trying to predict each and every test set comments but i get this error :

    499/499 [==============================] - 126s 253ms/step
    ---------------------------------------------------------------------------
    ValueError                                Traceback (most recent call last)
    <ipython-input-23-1dc84858379e> in <module>
    ----> 1 sub['toxic'] = model.predict(test_dataset, verbose=1)
          2 sub.to_csv('submission.csv', index=False)

    /opt/conda/lib/python3.7/site-packages/pandas/core/frame.py in __setitem__(self, key, value)
       2936         else:
       2937             # set column
    -> 2938             self._set_item(key, value)
       2939
       2940     def _setitem_slice(self, key, value):

    /opt/conda/lib/python3.7/site-packages/pandas/core/frame.py in _set_item(self, key, value)
       2998
       2999         self._ensure_valid_index(value)
    -> 3000         value = self._sanitize_column(key, value)
       3001         NDFrame._set_item(self, key, value)
       3002

    /opt/conda/lib/python3.7/site-packages/pandas/core/frame.py in _sanitize_column(self, key, value, broadcast)
       3634
       3635             # turn me into an ndarray
    -> 3636             value = sanitize_index(value, self.index, copy=False)
       3637             if not isinstance(value, (np.ndarray, Index)):
       3638                 if isinstance(value, list) and len(value) > 0:

    /opt/conda/lib/python3.7/site-packages/pandas/core/internals/construction.py in sanitize_index(data, index, copy)
        609
        610     if len(data) != len(index):
    --> 611         raise ValueError("Length of values does not match length of index")
        612
        613     if isinstance(data, ABCIndexClass) and not copy:

    ValueError: Length of values does not match length of index

now i have 4 questions :

1. is my implementation correct?

2. why i am getting that error? if i treat this problem as simple multilingual classification task like compute 1 loss for y true then i get no error at all,so where i am having trouble?

3. how can i solve the issue?

4. as it is my first time with multi task learning using huggingface transformers,what are your suggestions for updating my model so that it can generalize better?

  [1]: https://arxiv.org/abs/1706.05098
  [2]: https://huggingface.co/jplu/tf-xlm-roberta-large


ImportError: cannot import name 'MODEL_WITH_LM_HEAD_MAPPING' from 'transformers' (C:\Users\<username>\anaconda3\envs\tensorflow\lib\site-packages\transformers\__init__.py)

Model I am using (Bert, XLNet ...): GPT2

Language I am using the model on (English, Chinese ...): English

The problem arises when using:
[x] the official example scripts: (give details below)
a necessary package does not exist for import

The tasks I am working on is:
[x] my own task or dataset: (give details below)
I am trying to finetune the GPT2 library to work with recipes

## To reproduce

Steps to reproduce the behavior:

1. create conda virtualenv
2. install latest version of Tensorflow GPU
3. install transformers library
4. navigate to the 'language-modeling' folder
5. run 'python run_language_modeling.py' in conda virtualenv

LOG:
```
python run_language_modeling.py
2020-06-01 23:53:43.603759: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cudart64_101.dll
Traceback (most recent call last):
  File "run_language_modeling.py", line 29, in <module>
    from transformers import (
ImportError: cannot import name 'MODEL_WITH_LM_HEAD_MAPPING' from 'transformers' (C:\Users\<name>\anaconda3\envs\tensorflow\lib\site-packages\transformers\__init__.py)
```

## Expected behavior

There should be an error with not receiving variables or no output at all. If you were to insert data it would train an instance of GPT2 and save it to the output directory

## Environment info

- `transformers` version: 2.10.0
- Platform: Windows-10-10.0.18362-SP0
- Python version: 3.7.7
- PyTorch version (GPU?): not installed (NA)
- Tensorflow version (GPU): 2.1.0 (True)
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: No

It would be nice to figure out what the issue is. I am aware that there are 3 questions similar to this, but they all use torch while I am using tensorflow. All of the solutions I have tried (reinstalling tensorflow, reinstalling transformers, updating packages, reformatting code, etc.) have not worked. Thanks.


Is there any need to fine-tune the already pre-trained GPT-2 models?

Hello,

If I am using the pre-trained GPT-2 for my research, should I still fine-tune the already pre-trained models with my dataset? I am a bit confused by the term "pre-trained".

Thank you,


Converting model to pytorch

# 🐛 Bug
Folks, I am trying to convert the Biobert model to Pytorch. Here are the things that I did so far:

**1. For the vocab:**  I am trying to convert the vocab using solution from #69 :
```tokenizer = BartTokenizer.from_pretrained('/content/biobert_v1.1_pubmed/vocab.txt')```

I get :
`OSError: Model name '/content/biobert_v1.1_pubmed' was not found in tokenizers model name list (bart-large, bart-large-mnli, bart-large-cnn, bart-large-xsum). We assumed '/content/biobert_v1.1_pubmed' was a path, a model identifier, or url to a directory containing vocabulary files named ['vocab.json', 'merges.txt'] but couldn't find such vocabulary files at this path or url.`

I don’t have the vocab.json, so I how do I convert the vocab for the tokenizer ?

**2. For the model:** As the out of the box `pytorch_pretrained_bert.convert_tf_checkpoint_to_pytorch` did not work I customized it per #2 by adding:

```
excluded = ['BERTAdam','_power','global_step']
init_vars = list(filter(lambda x:all([True if e not in x[0] else False for e in excluded]),init_vars))

```
With this the model 'seems' to be converting fine.  But When I load this using:

`model = BartForConditionalGeneration.from_pretrained('path/to/model/biobert_v1.1_pubmed_pytorch.model') `

I still get

`UnicodeDecodeError: 'utf-8' codec can't decode byte 0x80 in position 0: invalid start byte
`

Can you pl. help me to understand what is going on here ?


Make docstring match args

When replying to #4698, I realized some language model docstrings are using arguments that are not present in the function signature. This PR addresses that (for all the ones I found at least).

The alternative would be to change the argument names in the function signatures (if it makes the various model APIs more consistent).


Specify PyTorch versions

Specify that the examples require a different PyTorch version than the base library.


Wrong argument passed during TFRobertaClassificationHead initialization

Core: Modeling

Internals of the library; Models.

Should Fix

This has been identified as a bug and should be fixed.

TensorFlow

Anything TensorFlow

# 🐛 Bug

## Information

There is an issue preventing a RoBERTa classification model from being serialized. It is related to a problem in passing `config` as the first argument to `tf.keras.layers.Layer`. However, the [expected positional argument](https://www.tensorflow.org/api_docs/python/tf/keras/layers/Layer) is `trainable`:

https://github.com/huggingface/transformers/blob/d6a677b14bcfd56b22fafeb212a27c6068886e07/src/transformers/modeling_tf_roberta.py#L327-L331

This is the root cause behind issue #3664 (about serialization).
A related fix for GPT2: #2738.

Model I am using (Bert, XLNet ...):
RoBERTa

Language I am using the model on (English, Chinese ...):
English
The problem arises when using:
* [ ] the official example scripts: (give details below)
* [x] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [x] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:

1. Run the code below:
```python
from transformers import (TFRobertaForSequenceClassification)
base_model = TFRobertaForSequenceClassification.from_pretrained("roberta-base")
print(base_model.classifier.trainable)
```

## Expected behavior
The output is:
`True`

The current output is
```
RobertaConfig {
  "architectures": [
    "RobertaForMaskedLM"
  ],
  "attention_probs_dropout_prob": 0.1,
  "bos_token_id": 0,
  "eos_token_id": 2,
  "hidden_act": "gelu",
  "hidden_dropout_prob": 0.1,
  "hidden_size": 768,
  "initializer_range": 0.02,
  "intermediate_size": 3072,
  "layer_norm_eps": 1e-05,
  "max_position_embeddings": 514,
  "model_type": "roberta",
  "num_attention_heads": 12,
  "num_hidden_layers": 12,
  "pad_token_id": 1,
  "type_vocab_size": 1,
  "vocab_size": 50265
}
```

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 2.10.0
- Platform: Colab
- Python version: 3.6.9
- PyTorch version (GPU?):
- Tensorflow version (GPU?): 2.2.0
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: No


Is the separation token absolutely necessary if I use GPT2DoubleHeadsModel with token_type_ids?

Hello,

I am trying to use the GPT2DoubleHeadsModel to process the multiple choice questions.
For the pre-processing of the multiple choice questions, I didn't add any special separating token between the multiple choice question and the multiple choice option. Instead, I generated the token_type_ids, which denote 0 for the question portion of the text, and 1 for the multiple choice option. Then I tried to make the GPT2DoubleHeadsModel to predict the correct answer by doing:

```python
gpt2DoubleHeadsModel(input_ids=input_ids, token_type_ids = token_type_ids)
```

Is this practice acceptable? or do I absolutely need to insert a special separation token between the question text and the multiple choice option text?

Thank you,


How tokenizers work

Good afternoon.
If possible, I would ask a few questions which I cannot solve for several days running.

1) What does the file _merge.txt_ mean which is used in `ByteLevelBPETokenizer`? Does it store the config for the tokenizer? I deepened into it, however, did not manage to understand its purpose.
2) Is it possible to get offsets for the text when using models-stacked tokenizers? For instance, I am using ElectraTokenizer and I would like it to return the offsets for my texts - do you probably have any templates of how to do it?

Thanks in advance!


When using the Hugging Face Transformer, if I set my pad_token to be a different token then the default, do I need to train my model on that new pad_token as well?

Hello,

For using the GPT2DoubleHeadsModel, I used ``<eos>`` as the last token of my text sequence, which my model will use to make predictions for multiple-choice questions.
I also set ``<pad>`` as the padding token in my tokenizer, which is different than the default.

When using the Hugging Face pre-trained GPT2DoubleHeadsModel, do I need to train the already pre-trained Transformer because the two tokens I mentioned above are new?

Thank you,


Is transformers 2.11.0 compatible with tokenizers 0.8.0(-dev*)?


Tensorflow XLMRoberta Multi-Class Problem

## Details

I am attempting to fine tune an XLMRoberta sequence classification model. I have an array of text snippets from physicians labelled 1-8 with various diagnostic indications. I've created a tensorflow dataset object with the
```
convert_raw_to_xlmroberta_tfdataset()
```
 function seen here (https://stackoverflow.com/questions/62095316/tensorflow-xlmroberta-multi-class)

I then create the model:
```
from transformers import TFXLMRobertaForSequenceClassification
import tensorflow as tf

learning_rate = 2e-5

number_of_epochs = 2
model = TFXLMRobertaForSequenceClassification.from_pretrained("jplu/tf-xlm-roberta-base")
optimizer = tf.keras.optimizers.Adam(learning_rate=learning_rate, epsilon=1e-08)
loss = tf.keras.losses.SparseCategoricalCrossentropy()
metric = tf.keras.metrics.SparseCategoricalAccuracy('accuracy')
model.compile(optimizer=optimizer, loss=loss, metrics=[metric])
```
but consistently get this error:
```
ValueError: Shapes (None, 1, 8) and (None, 2) are incompatible
```
(full trace at the above SO link).

I've tried both Sparse Categorical Cross Entropy and just Categorical Cross Entropy. I've used one-hot encoded labels and "normal" labels. Is it even possible to do multi-class classification with TFXLMRoberta? It started to work when I fed in a binary dummy set of labels.


Fused_norm_layer_cuda

Hi,
I have created a model on kaggle tpu using run_pretraining.py and it gives me tf_checkpoint file. First of all the model is 12gb in size due to which it is throwing memory error while loading weights into it on kaggle and colab.
Is there a way to reduce the size?

Secondly, I tried performing the operation over a linux server without Gpu so it throws an error no module "fused_norm_layer_cuda" which is obvious but I want to know if there is a way to convert the model from tf to pytorch without GPU, or is there any parameter in BertForPreTraining which can instantiate the model without GPU. Please help. 🤗


Why DataCollatorForLanguageModeling do not make attention_mask feature?

# ❓ Questions & Help

<!-- The GitHub issue tracker is primarly intended for bugs, feature requests,
     new models and benchmarks, and migration questions. For all other questions,
     we direct you to Stack Overflow (SO) where a whole community of PyTorch and
     Tensorflow enthusiast can help you out. Make sure to tag your question with the
     right deep learning framework as well as the huggingface-transformers tag:
     https://stackoverflow.com/questions/tagged/huggingface-transformers

     If your question wasn't answered after a period of time on Stack Overflow, you
     can always open a question on GitHub. You should then link to the SO question
     that you posted.
     -->

## Details
<!-- Description of your issue -->
As we know, according to padding on batch of input_ids, we should mask the padding ids as invalidate to avoid attention on them. But I found that collate function of DataCollatorForLanguageModeling didn't implement attention_mask feature.
Is this expected? Or do you think attention_mask is not necessary?

<!-- You should first ask your question on SO, and only if
     you didn't get an answer ask it here on GitHub. -->
**A link to original question on Stack Overflow**:


Why we need the init_weight function in BERT pretrained model

Usage

General questions about the library

# ❓ Questions & Help

I have already tried asking the question is SO, which you can find the link [here](https://stackoverflow.com/questions/62040309/why-we-need-the-init-weight-function-in-bert-pretrained-model-in-huggingface-tra/62053791#62053791).

## Details
In the code by Hugginface transformers, there are many fine-tuning models have the function `init_weight`.
For example([here](https://github.com/huggingface/transformers/blob/a9aa7456ac/src/transformers/modeling_bert.py#L1073-L1082)), there is a `init_weight` function at last. Even though we use `from_pretrained`, it will still call the constructor and call `init_weight` function.
```python
class BertForSequenceClassification(BertPreTrainedModel):
    def __init__(self, config):
        super().__init__(config)
        self.num_labels = config.num_labels

        self.bert = BertModel(config)
        self.dropout = nn.Dropout(config.hidden_dropout_prob)
        self.classifier = nn.Linear(config.hidden_size, config.num_labels)

        self.init_weights()

```
As I know, it will call the following [code](https://github.com/huggingface/transformers/blob/a9aa7456ac/src/transformers/modeling_bert.py#L520-L530)
```python
def _init_weights(self, module):
    """ Initialize the weights """
    if isinstance(module, (nn.Linear, nn.Embedding)):
        # Slightly different from the TF version which uses truncated_normal for initialization
        # cf https://github.com/pytorch/pytorch/pull/5617
        module.weight.data.normal_(mean=0.0, std=self.config.initializer_range)
    elif isinstance(module, BertLayerNorm):
        module.bias.data.zero_()
        module.weight.data.fill_(1.0)
    if isinstance(module, nn.Linear) and module.bias is not None:
        module.bias.data.zero_()
```

My question is **If we are loading the pre-trained language model, why do we need to initialize the weight for every module?**

I guess I must be misunderstanding something here.


Add community notebook for T5 sentiment span extraction

This is an example notebook that aims to increase the coverage of T5 fine-tuning examples to address #4426 .

This notebook presents a high level overview of T5, its significance for the future of NLP in practice, and a thoroughly commented tutorial on how to fine-tune T5 for sentiment span extraction with an extractive Q&A format.

I recently presented this in a webinar published on [youtube](https://www.youtube.com/watch?v=4LYw_UIdd4A).


NER example doesn’t work with tensorflow

TensorFlow

Anything TensorFlow

I’m working through the Pytorch text tokenization example here using the tensorflow version run_tf_ner.py: https://github.com/huggingface/transformers/tree/master/examples/token-classification

The PyTorch version using run_ner.py works. I believe the difference is in https://github.com/huggingface/transformers/blob/master/examples/token-classification/utils_ner.py starting at line 139 where it has different logic for tensorflow. Narrowing down to utils_ner.py line 149:

`pad_token_label_id: int = -1`

The Pytorch version uses -100. This seems to be the only labeling difference that I can tell. I tried to change this line to -100 as well. But tensorflow code doesn’t seem to accept -1 or -100.

Anyone know how to get this example to work using the tensorflow version? Thanks!

I created a Colab for the example here: https://colab.research.google.com/drive/10GrYSYx5sUVMXplgUS79fIV73bFtoiHX?usp=sharing

This is the error I keep getting:

```
05/31/2020 00:07:45 - INFO - utils_ner -   tokens: [CLS] dar ##aus en ##t ##wick ##elt ##e sic ##h im ro ##ko ##ko die sit ##te des gem ##ein ##sam ##en wei ##nen ##s im theater , das die stand ##es ##gren ##zen inner ##hal ##b des pub ##lik ##ums uber ##bruck ##en sol ##lt ##e . [SEP]
05/31/2020 00:07:45 - INFO - utils_ner -   input_ids: 101 18243 20559 4372 2102 7184 20042 2063 14387 2232 10047 20996 3683 3683 3280 4133 2618 4078 17070 12377 21559 2368 11417 10224 2015 10047 4258 1010 8695 3280 3233 2229 13565 10431 5110 8865 2497 4078 9047 18393 18163 19169 28985 2368 14017 7096 2063 1012 102 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
05/31/2020 00:07:45 - INFO - utils_ner -   input_mask: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
05/31/2020 00:07:45 - INFO - utils_ner -   segment_ids: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
05/31/2020 00:07:45 - INFO - utils_ner -   label_ids: -1 24 -1 24 -1 -1 -1 -1 24 -1 24 6 -1 -1 24 24 -1 24 24 -1 -1 -1 24 -1 -1 24 24 24 24 24 24 -1 -1 -1 24 -1 -1 24 24 -1 -1 24 -1 -1 24 -1 -1 24 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1
05/31/2020 00:07:54 - WARNING - transformers.training_args -   Using deprecated `--per_gpu_train_batch_size` argument which will be removed in a future version. Using `--per_device_train_batch_size` is preferred.
05/31/2020 00:07:54 - WARNING - transformers.training_args -   Using deprecated `--per_gpu_train_batch_size` argument which will be removed in a future version. Using `--per_device_train_batch_size` is preferred.
05/31/2020 00:07:54 - INFO - transformers.trainer_tf -   Created an/a adam optimizer
05/31/2020 00:07:54 - INFO - transformers.trainer_tf -   ***** Running training *****
05/31/2020 00:07:54 - INFO - transformers.trainer_tf -     Num examples = 24000
05/31/2020 00:07:54 - INFO - transformers.trainer_tf -     Num Epochs = 3
05/31/2020 00:07:54 - INFO - transformers.trainer_tf -     Total optimization steps = 750
WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/transformers/trainer_tf.py:360: StrategyBase.experimental_run_v2 (from tensorflow.python.distribute.distribute_lib) is deprecated and will be removed in a future version.
Instructions for updating:
renamed to `run`
05/31/2020 00:07:54 - WARNING - tensorflow -   From /usr/local/lib/python3.6/dist-packages/transformers/trainer_tf.py:360: StrategyBase.experimental_run_v2 (from tensorflow.python.distribute.distribute_lib) is deprecated and will be removed in a future version.
Instructions for updating:
renamed to `run`
WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensorflow/python/ops/resource_variable_ops.py:1817: calling BaseResourceVariable.__init__ (from tensorflow.python.ops.resource_variable_ops) with constraint is deprecated and will be removed in a future version.
Instructions for updating:
If using Keras pass *_constraint arguments to layers.
05/31/2020 00:08:02 - WARNING - tensorflow -   From /usr/local/lib/python3.6/dist-packages/tensorflow/python/ops/resource_variable_ops.py:1817: calling BaseResourceVariable.__init__ (from tensorflow.python.ops.resource_variable_ops) with constraint is deprecated and will be removed in a future version.
Instructions for updating:
If using Keras pass *_constraint arguments to layers.
/usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/indexed_slices.py:434: UserWarning: Converting sparse IndexedSlices to a dense Tensor of unknown shape. This may consume a large amount of memory.
  "Converting sparse IndexedSlices to a dense Tensor of unknown shape. "
2020-05-31 00:08:52.364556: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at sparse_xent_op.cc:90 : Invalid argument: Received a label value of -1 which is outside the valid range of [0, 25).  Label values: -1 24 -1 -1 -1 -1 24 24 -1 -1 -1 24 -1 -1 -1 -1 -1 -1 -1 24 24 2 -1 -1 -1 24 0 24 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 24 24 -1 -1 -1 -1 24 24 -1 -1 24 -1 -1 -1 24 -1 -1 24 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 24 24 -1 -1 -1 -1 -1 24 -1 24 -1 24 -1 24 -1 -1 24 -1 24 24 6 24 -1 24 -1 24 -1 24 -1 -1 -1 24 -1 -1 -1 24 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 0 24 3 -1 24 24 24 24 -1 -1 1 24 -1 -1 -1 -1 9 21 -1 -1 24 0 -1 -1 -1 24 24 24 24 -1 -1 24 24 -1 -1 24 24 -1 -1 -1 3 15 15 24 -1 -1 -1 24 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 24 24 24 24 24 -1 -1 24 -1 24 24 -1 -1 24 -1 24 -1 -1 -1 -1 24 24 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 24 6 24 24 24 6 18 18 24 24 6 24 -1 -1 24 24 24 -1 -1 -1 24 24 9 21 -1 24 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 24 -1 -1 -1 24 -1 24 -1 24 -1 24 -1 24 -1 24 24 24 -1 -1 24 24 24 -1 24 -1 24 -1 24 -1 -1 -1 -1 24 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 24 24 -1 24 -1 -1 -1 24 -1 -1 -1 24 -1 9 24 9 21 24 24 -1 24 24 0 -1 -1 24 -1 -1 24 24 -1 24 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 24 -1 24 -1 24 -1 24 24 -1 -1 -1 24 -1 24 24 -1 24 -1 24 -1 24 -1 24 -1 -1 -1 24 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 24 24 -1 24 -1 -1 24 -1 0 -1 24 2 -1 -1 -1 24 24 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 24 24 24 -1 -1 -1 -1 -1 24 24 3 -1 -1 -1 -1 -1 -1 24 -1 24 -1 -1 24 24 -1 -1 24 24 24 -1 -1 -1 24 24 -1 -1 -1 -1 -1 24 -1 -1 24 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 24 -1 -1 24 24 24 -1 24 -1 24 -1 24 -1 24 24 -1 24 -1 -1 24 -1 24 24 24 24 -1 -1 -1 -1 24 24 -1 -1 -1 -1 24 -1 -1 24 -1 24 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 24 24 24 -1 24 24 -1 24 24 24 -1 -1 24 -1 -1 -1 24 -1 24 -1 -1 24 24 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 24 -1 24 -1 -1 -1 24 -1 -1 24 -1 24 -1 -1 24 24 -1 -1 24 -1 -1 -1 -1 24 -1 -1 24 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 24 -1 24 24 -1 24 -1 24 24 24 24 -1 9 24 -1 -1 24 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 24 24 24 -1 24 -1 24 24 -1 24 -1 24 -1 24 -1 24 24 24 24 -1 -1 24 -1 24 -1 24 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 24 -1 24 -1 -1 24 -1 24 -1 24 -1 -1 24 -1 -1 -1 -1 -1 -1 24 -1 24 24 -1 24 24 -1 -1 24 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 24 6 18 24 -1 -1 24 -1 -1 24 -1 -1 24 24 -1 -1 -1 -1 -1 24 -1 24 -1 -1 24 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 24 8 -1 -1 -1 -1 -1 -1 -1 24 -1 -1 24 -1 -1 -1 24 24 -1 -1 -1 8 -1 -1 -1 -1 -1 -1 -1 24 24 -1 24 -1 24 -1 24 24 -1 -1 -1 24 -1 -1 24 24 24 24 8 -1 -1 -1 -1 -1 -1 -1 24 -1 -1 -1 -1 24 24 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 24 24 6 -1 -1 18 18 -1 -1 -1 18 18 18 -1 18 18 -1 -1 -1 24 24 -1 -1 -1 -1 24 24 0 -1 24 -1 -1 -1 24 -1 -1 24 -1 -1 -1 24 24 -1 -1 24 24 -1 24 24 0 -1 24 -1 -1 24 24 -1 24 5 -1 -1 -1 -1 -1 -1 24 0 12 12 24 -1 -1 -1 24 -1 24 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 24 -1 -1 24 24 24 24 24 -1 -1 -1 -1 -1 24 -1 -1 -1 24 -1 -1 24 24 -1 24 24 24 24 -1 -1 24 24 -1 -1 -1 -1 0 -1 -1 -1 24 24 -1 -1 24 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 24 -1 -1 24 24 24 -1 24 -1 24 -1 6 -1 -1 24 24 1 -1 24 -1 -1 -1 -1 -1 -1 24 24 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 24 24 24 -1 -1 24 3 -1 -1 24 24 24 24 24 24 -1 24 -1 24 24 24 -1 24 -1 -1 -1 -1 24 24 -1 -1 -1 -1 24 24 -1 -1 -1 -1 -1 24 -1 -1 24 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 24 24 24 -1 24 24 -1 24 24 24 -1 -1 -1 24 24 -1 24 -1 24 -1 -1 -1 24 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 24 24 -1 24 -1 -1 -1 -1 -1 24 24 24 -1 -1 -1 -1 24 -1 24 -1 -1 24 24 3 15 15 -1 -1 -1 -1 -1 24 0 12 12 24 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 24 -1 -1 0 -1 -1 24 -1 -1 -1 24 -1 -1 -1 -1 -1 -1 -1 -1 24 24 24 -1 0 -1 -1 24 -1 24 -1 24 -1 24 -1 -1 24 24 -1 -1 -1 24 24 -1 -1 24 -1 -1 -1 -1 24 -1 -1 -1 24 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 24 24 -1 -1 9 21 -1 24 9 -1 21 24 9 21 24 24 -1 -1 -1 -1 24 -1 -1 -1 -1 24 -1 -1 24 24 -1 24 -1 24 24 -1 24 24 -1 -1 24 24 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 24 24 -1 24 24 -1 24 -1 24 -1 24 24 1 -1 -1 24 24 24 -1 24 -1 -1 24 24 -1 -1 -1 -1 -1 24 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 9 -1 24 -1 -1 -1 -1 24 -1 24 24 -1 24 24 9 24 24 24 24 -1 -1 -1 -1 24 24 -1 -1 -1 24 0 24 24 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 24 9 -1 24 24 -1 24 24 -1 24 -1 -1 -1 -1 24 -1 -1 -1 24 -1 -1 -1 -1 -1 24 24 24 -1 24 -1 -1 24 24 -1 24 -1 -1 -1 24 24 6 -1 -1 24 24 -1 24 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 24 24 -1 -1 24 -1 -1 -1 -1 24 -1 24 24 -1 -1 -1 -1 -1 24 24 -1 -1 -1 -1 24 -1 -1 24 24 -1 24 24 -1 24 24 -1 -1 -1 24 -1 -1 -1 -1 24 -1 24 -1 24 24 -1 -1 24 24 -1 -1 24 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 24 -1 -1 24 24 -1 -1 -1 -1 24 -1 -1 24 -1 -1 -1 -1 -1 -1 24 -1 24 -1 -1 24 24 24 -1 -1 -1 -1 -1 24 24 24 -1 -1 -1 24 -1 -1 24 -1 -1 24 -1 24 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1
Traceback (most recent call last):
  File "run_tf_ner.py", line 281, in <module>
    main()
  File "run_tf_ner.py", line 213, in main
    trainer.train()
  File "/usr/local/lib/python3.6/dist-packages/transformers/trainer_tf.py", line 277, in train
    for training_loss in self._training_steps():
  File "/usr/local/lib/python3.6/dist-packages/transformers/trainer_tf.py", line 321, in _training_steps
    for i, loss in enumerate(self._accumulate_next_gradients()):
  File "/usr/local/lib/python3.6/dist-packages/transformers/trainer_tf.py", line 354, in _accumulate_next_gradients
    yield _accumulate_next()
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/eager/def_function.py", line 580, in __call__
    result = self._call(*args, **kwds)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/eager/def_function.py", line 708, in _call
    return function_lib.defun(fn_with_cond)(*canon_args, **canon_kwds)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/eager/function.py", line 2420, in __call__
    return graph_function._filtered_call(args, kwargs)  # pylint: disable=protected-access
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/eager/function.py", line 1665, in _filtered_call
    self.captured_inputs)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/eager/function.py", line 1746, in _call_flat
    ctx, args, cancellation_manager=cancellation_manager))
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/eager/function.py", line 598, in call
    ctx=ctx)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/eager/execute.py", line 60, in quick_execute
    inputs, attrs, num_outputs)
tensorflow.python.framework.errors_impl.InvalidArgumentError:  Received a label value of -1 which is outside the valid range of [0, 25).  Label values: -1 24 -1 -1 -1 -1 24 24 -1 -1 -1 24 -1 -1 -1 -1 -1 -1 -1 24 24 2 -1 -1 -1 24 0 24 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 24 24 -1 -1 -1 -1 24 24 -1 -1 24 -1 -1 -1 24 -1 -1 24 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 24 24 -1 -1 -1 -1 -1 24 -1 24 -1 24 -1 24 -1 -1 24 -1 24 24 6 24 -1 24 -1 24 -1 24 -1 -1 -1 24 -1 -1 -1 24 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 0 24 3 -1 24 24 24 24 -1 -1 1 24 -1 -1 -1 -1 9 21 -1 -1 24 0 -1 -1 -1 24 24 24 24 -1 -1 24 24 -1 -1 24 24 -1 -1 -1 3 15 15 24 -1 -1 -1 24 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 24 24 24 24 24 -1 -1 24 -1 24 24 -1 -1 24 -1 24 -1 -1 -1 -1 24 24 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 24 6 24 24 24 6 18 18 24 24 6 24 -1 -1 24 24 24 -1 -1 -1 24 24 9 21 -1 24 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 24 -1 -1 -1 24 -1 24 -1 24 -1 24 -1 24 -1 24 24 24 -1 -1 24 24 24 -1 24 -1 24 -1 24 -1 -1 -1 -1 24 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 24 24 -1 24 -1 -1 -1 24 -1 -1 -1 24 -1 9 24 9 21 24 24 -1 24 24 0 -1 -1 24 -1 -1 24 24 -1 24 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 24 -1 24 -1 24 -1 24 24 -1 -1 -1 24 -1 24 24 -1 24 -1 24 -1 24 -1 24 -1 -1 -1 24 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 24 24 -1 24 -1 -1 24 -1 0 -1 24 2 -1 -1 -1 24 24 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 24 24 24 -1 -1 -1 -1 -1 24 24 3 -1 -1 -1 -1 -1 -1 24 -1 24 -1 -1 24 24 -1 -1 24 24 24 -1 -1 -1 24 24 -1 -1 -1 -1 -1 24 -1 -1 24 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 24 -1 -1 24 24 24 -1 24 -1 24 -1 24 -1 24 24 -1 24 -1 -1 24 -1 24 24 24 24 -1 -1 -1 -1 24 24 -1 -1 -1 -1 24 -1 -1 24 -1 24 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 24 24 24 -1 24 24 -1 24 24 24 -1 -1 24 -1 -1 -1 24 -1 24 -1 -1 24 24 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 24 -1 24 -1 -1 -1 24 -1 -1 24 -1 24 -1 -1 24 24 -1 -1 24 -1 -1 -1 -1 24 -1 -1 24 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 24 -1 24 24 -1 24 -1 24 24 24 24 -1 9 24 -1 -1 24 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 24 24 24 -1 24 -1 24 24 -1 24 -1 24 -1 24 -1 24 24 24 24 -1 -1 24 -1 24 -1 24 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 24 -1 24 -1 -1 24 -1 24 -1 24 -1 -1 24 -1 -1 -1 -1 -1 -1 24 -1 24 24 -1 24 24 -1 -1 24 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 24 6 18 24 -1 -1 24 -1 -1 24 -1 -1 24 24 -1 -1 -1 -1 -1 24 -1 24 -1 -1 24 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 24 8 -1 -1 -1 -1 -1 -1 -1 24 -1 -1 24 -1 -1 -1 24 24 -1 -1 -1 8 -1 -1 -1 -1 -1 -1 -1 24 24 -1 24 -1 24 -1 24 24 -1 -1 -1 24 -1 -1 24 24 24 24 8 -1 -1 -1 -1 -1 -1 -1 24 -1 -1 -1 -1 24 24 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 24 24 6 -1 -1 18 18 -1 -1 -1 18 18 18 -1 18 18 -1 -1 -1 24 24 -1 -1 -1 -1 24 24 0 -1 24 -1 -1 -1 24 -1 -1 24 -1 -1 -1 24 24 -1 -1 24 24 -1 24 24 0 -1 24 -1 -1 24 24 -1 24 5 -1 -1 -1 -1 -1 -1 24 0 12 12 24 -1 -1 -1 24 -1 24 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 24 -1 -1 24 24 24 24 24 -1 -1 -1 -1 -1 24 -1 -1 -1 24 -1 -1 24 24 -1 24 24 24 24 -1 -1 24 24 -1 -1 -1 -1 0 -1 -1 -1 24 24 -1 -1 24 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 24 -1 -1 24 24 24 -1 24 -1 24 -1 6 -1 -1 24 24 1 -1 24 -1 -1 -1 -1 -1 -1 24 24 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 24 24 24 -1 -1 24 3 -1 -1 24 24 24 24 24 24 -1 24 -1 24 24 24 -1 24 -1 -1 -1 -1 24 24 -1 -1 -1 -1 24 24 -1 -1 -1 -1 -1 24 -1 -1 24 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 24 24 24 -1 24 24 -1 24 24 24 -1 -1 -1 24 24 -1 24 -1 24 -1 -1 -1 24 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 24 24 -1 24 -1 -1 -1 -1 -1 24 24 24 -1 -1 -1 -1 24 -1 24 -1 -1 24 24 3 15 15 -1 -1 -1 -1 -1 24 0 12 12 24 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 24 -1 -1 0 -1 -1 24 -1 -1 -1 24 -1 -1 -1 -1 -1 -1 -1 -1 24 24 24 -1 0 -1 -1 24 -1 24 -1 24 -1 24 -1 -1 24 24 -1 -1 -1 24 24 -1 -1 24 -1 -1 -1 -1 24 -1 -1 -1 24 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 24 24 -1 -1 9 21 -1 24 9 -1 21 24 9 21 24 24 -1 -1 -1 -1 24 -1 -1 -1 -1 24 -1 -1 24 24 -1 24 -1 24 24 -1 24 24 -1 -1 24 24 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 24 24 -1 24 24 -1 24 -1 24 -1 24 24 1 -1 -1 24 24 24 -1 24 -1 -1 24 24 -1 -1 -1 -1 -1 24 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 9 -1 24 -1 -1 -1 -1 24 -1 24 24 -1 24 24 9 24 24 24 24 -1 -1 -1 -1 24 24 -1 -1 -1 24 0 24 24 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 24 9 -1 24 24 -1 24 24 -1 24 -1 -1 -1 -1 24 -1 -1 -1 24 -1 -1 -1 -1 -1 24 24 24 -1 24 -1 -1 24 24 -1 24 -1 -1 -1 24 24 6 -1 -1 24 24 -1 24 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 24 24 -1 -1 24 -1 -1 -1 -1 24 -1 24 24 -1 -1 -1 -1 -1 24 24 -1 -1 -1 -1 24 -1 -1 24 24 -1 24 24 -1 24 24 -1 -1 -1 24 -1 -1 -1 -1 24 -1 24 -1 24 24 -1 -1 24 24 -1 -1 24 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 24 -1 -1 24 24 -1 -1 -1 -1 24 -1 -1 24 -1 -1 -1 -1 -1 -1 24 -1 24 -1 -1 24 24 24 -1 -1 -1 -1 -1 24 24 24 -1 -1 -1 24 -1 -1 24 -1 -1 24 -1 24 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1
	 [[{{node cond/else/_1/StatefulPartitionedCall/sparse_categorical_crossentropy/SparseSoftmaxCrossEntropyWithLogits/SparseSoftmaxCrossEntropyWithLogits}}]] [Op:__inference_fn_with_cond_30948]

Function call stack:
fn_with_cond
```


Transformer-XL: Input and labels for Language Modeling

# ❓ Questions & Help

<!-- The GitHub issue tracker is primarly intended for bugs, feature requests,
     new models and benchmarks, and migration questions. For all other questions,
     we direct you to Stack Overflow (SO) where a whole community of PyTorch and
     Tensorflow enthusiast can help you out. Make sure to tag your question with the
     right deep learning framework as well as the huggingface-transformers tag:
     https://stackoverflow.com/questions/tagged/huggingface-transformers

     If your question wasn't answered after a period of time on Stack Overflow, you
     can always open a question on GitHub. You should then link to the SO question
     that you posted.
     -->

## Details
<!-- Description of your issue -->
I'm trying to finetune the pretrained Transformer-XL model `transfo-xl-wt103` for a language modeling task. Therfore, I use the model class `TransfoXLLMHeadModel`.

To iterate over my dataset I use the `LMOrderedIterator` from the file [tokenization_transfo_xl.py](https://github.com/huggingface/transformers/blob/5e737018e1fcb22c8b76052058279552a8d6c806/src/transformers/tokenization_transfo_xl.py#L467) which yields a tensor with the `data` and its `target` for each batch (and the sequence length).

**My question**:
Let's assume the following data with `batch_size = 1` and `bptt = 8`:

    data = tensor([[1,2,3,4,5,6,7,8]])
    target = tensor([[2,3,4,5,6,7,8,9]])
    mems # from the previous output

I currently pass this data into the model like this:

    output = model(input_ids=data, labels=target, mems=mems)

Is this correct?

I am wondering because the documentation says for the `labels` parameter:

> labels (:obj:`torch.LongTensor` of shape :obj:`(batch_size, sequence_length)`, `optional`, defaults to :obj:`None`):
            Labels for language modeling.
            Note that the labels **are shifted** inside the model, i.e. you can set ``lm_labels = input_ids``

So what is it about the parameter `lm_labels`? I only see `labels` defined in the `forward` method.

And when the labels "are shifted" inside the model, does this mean I have to pass in `data` twice (for `input_ids` and `labels`) because `labels` shifted inside? But how does the model then know the next token to predict (in the case above: `9`) ?

I also read through [this bug](https://github.com/huggingface/transformers/issues/3711) and the fix in [this pull request](https://github.com/huggingface/transformers/pull/3716) but I don't quite understand how to treat the model now (before vs. after fix). Maybe someone could explain me both versions.

Thanks in advance for some help!

<!-- You should first ask your question on SO, and only if
     you didn't get an answer ask it here on GitHub. -->
**A link to original question on Stack Overflow**: https://stackoverflow.com/q/62069350/9478384


SpanBert always predicts the same token

I tried to use this implementation - https://huggingface.co/SpanBERT/spanbert-base-cased, but as a prediction I always get the same exect outoput no matter where I put [MASK] in a sentence. Here is the code.  Am I doing something wrong?

    import torch
    from transformers import AutoTokenizer,AutoModel
    tokenizer = AutoTokenizer.from_pretrained("SpanBERT/spanbert-base-cased")
    model = AutoModel.from_pretrained("SpanBERT/spanbert-base-cased")
    model.eval()
    model.to('cuda')

    text = "[CLS] Who was Jim Henson ? [SEP] Jim Henson was a puppeteer [SEP]"
    tokenized_text = tokenizer.tokenize(text)
    print("tokenized_text",tokenized_text)

    themask = 2
    tokenized_text[themask] = '[MASK]'

    indexes = tokenizer.convert_tokens_to_ids(tokenized_text)
    indexes_tensor = torch.tensor([indexes])
    indexes_tensor = indexes_tensor.to('cuda')

    with torch.no_grad():
        outputs = model(indexes_tensor)
        predictions0 = outputs[0]

    the_index = torch.argmax(predictions0[0, themask]).item()
    theresult = tokenizer.convert_ids_to_tokens([the_index])[0]
    print("theresult",theresult)
    print("the_index",the_index)


Loading config file bug

Hi guys.

I have released a new (XLNet based) transformer model for low-resource language Tigrinya
 [(TigXLNet)](https://github.com/abryeemessi/Transferring-Monolingual-Model-to-Low-Resource-Language)  and found a bug when loading a pre-trained config file:

My config file looks like:
https://s3.amazonaws.com/models.huggingface.co/bert/abryee/TigXLNet/config.json

       config = AutoConfig.from_pretrained("abryee/TigXLNet")
       print(config.d_head) #prints 48 even though d_head in the given config file is 64.


Please add the functionality to save tokenizer model for run_language_modeling.py

# 🚀 Feature request

Please add the feature to save the tokenizer model during training to the checkpoint folders.
## Motivation

When I tried out the script for [fine-tuning with language modeling](transformers/examples/language_modeling/run_language_modeling.py), I realized that the generated checkpoints during training cannot allow continue training, because under the checkpoint folders, the corresponding tokenizer model is not saved (including the files: tokenizer_config.json, special_tokens_map.json, vocab.txt). As I checked the [script](https://github.com/huggingface/transformers/blob/0866669e751bef636fa693b704a28c1fea9a17f3/examples/language-modeling/run_language_modeling.py#L250), and noticed that the tokenizer model is only saved after the training process.
## Your contribution

Haven't looked into the code in details so it might be best to have someone familiar with the Trainer class to integrate this.


Adding Neutral Score

# 🚀 Feature request

After performing some experimentation and comparison to VADER, we come to consensus that "pretrained BERT-based Hugging Face transfomer" is performing way beyond the other lexicons, but VADER is also good at social media context + it provides "neutral" label which turns out to be useful in some context.

I was wondering whether it is possible to manipulate the Transformer Sentiment Analysis in a way that it can calculate the **"neutral" score**?


# 🐛 Bug

## Information

Model I am using (Bert, XLNet ...):

Language I am using the model on (English, Chinese ...):

The problem arises when using:
* [ ] the official example scripts: (give details below)
* [ ] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [ ] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:
01-training-tokenizers.ipynb
```
# For the user's convenience `tokenizers` provides some very high-level classes encapsulating
# the overall pipeline for various well-known tokenization algorithm.
# Everything described below can be replaced by the ByteLevelBPETokenizer class.

from tokenizers import Tokenizer
from tokenizers.decoders import ByteLevel as ByteLevelDecoder
from tokenizers.models import BPE
from tokenizers.normalizers import Lowercase, NFKC, Sequence
from tokenizers.pre_tokenizers import ByteLevel

# First we create an empty Byte-Pair Encoding model (i.e. not trained model)
tokenizer = Tokenizer(BPE())

# Then we enable lower-casing and unicode-normalization
# The Sequence normalizer allows us to combine multiple Normalizer that will be
# executed in order.
tokenizer.normalizer = Sequence([
    NFKC(),
    Lowercase()
])

# Our tokenizer also needs a pre-tokenizer responsible for converting the input to a ByteLevel representation.
tokenizer.pre_tokenizer = ByteLevel()

# And finally, let's plug a decoder so we can recover from a tokenized input to the original one
tokenizer.decoder = ByteLevelDecoder()
```

<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

## Expected behavior

<!-- A clear and concise description of what you would expect to happen. -->
Flawless Run
## Environment info
- `transformers` version: 2.10.0
- Platform: Linux-5.4.0-31-generic-x86_64-with-debian-bullseye-sid
- Python version: 3.7.6
- PyTorch version (GPU?): 1.4.0 (False)
- Tensorflow version (GPU?): 2.1.0 (False)
- Using GPU in script?: <fill in>
- Using distributed or parallel set-up in script?: <fill in>

Specific recurring Error:
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-4-f099004b011b> in <module>
     10
     11 # First we create an empty Byte-Pair Encoding model (i.e. not trained model)
---> 12 tokenizer = Tokenizer(BPE())
     13
     14 # Then we enable lower-casing and unicode-normalization

TypeError: cannot create 'BPE' instances

NB. I have gone through this issue:https://github.com/huggingface/transformers/issues/3787,
It dosent solve it either.


Gradient overflow issue when i try to train gpt2 with run_language_modeling in fp16 with 02. Any idea why that maybe happen?

# ❓ Questions & Help

<!-- The GitHub issue tracker is primarly intended for bugs, feature requests,
     new models and benchmarks, and migration questions. For all other questions,
     we direct you to Stack Overflow (SO) where a whole community of PyTorch and
     Tensorflow enthusiast can help you out. Make sure to tag your question with the
     right deep learning framework as well as the huggingface-transformers tag:
     https://stackoverflow.com/questions/tagged/huggingface-transformers

     If your question wasn't answered after a period of time on Stack Overflow, you
     can always open a question on GitHub. You should then link to the SO question
     that you posted.
     -->

## Details
<!-- Description of your issue -->

<!-- You should first ask your question on SO, and only if
     you didn't get an answer ask it here on GitHub. -->
**A link to original question on Stack Overflow**:


[EncoderDecoder] Add RoBERTa as a decoder

* Add crossattention input
* Add EncoderDecoder tests for RoBERTa

Since RoBERTa is a subclass of BERT, it inherits all the crossattention mechanics in the model itself. This change allows RobertaForMaskedLM to take in encoder hidden states and language model labels to work with the EncoderDecoder framework.


Keyword errors on tokenizer.encode_plus

# 🐛 Bug

## Information

Model I am using (Bert, XLNet ...): BertTokenizer

Language I am using the model on (English, Chinese ...): English (`bert-base-uncased`)

The problem arises when using:
* [ ] the official example scripts: (give details below)
* [X ] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [X] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:

Code:
```
tokenizer = BERTTokenizer.from_pretrained('bert-base-uncased')`
result = tokenizer.encode_plus("This is an example sentence", add_special_tokens=True,
max_length=64, pad_to_max_length=True, return_attention_masks=True, return_tensors='pt'
```

Result:
```
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File ".../python3.8/site-packages/transformers/tokenization_utils.py", line 786, in encode_plus
        first_ids = get_input_ids(text)
File ".../python3.8/site-packages/transformers/tokenization_utils.py", line 778, in get_input_ids
        return self.convert_tokens_to_ids(self.tokenize(text, **kwargs))
File ".../python3.8/site-packages/transformers/tokenization_utils.py", line 649, in tokenize
        tokenized_text = split_on_tokens(added_tokens, text)
File ".../python3.8/site-packages/transformers/tokenization_utils.py", line 644, in split_on_tokens
        return sum((self._tokenize(token, **kwargs) if token not \
File ".../python3.8/site-packages/transformers/tokenization_utils.py", line 644, in <genexpr>
        return sum((self._tokenize(token, **kwargs) if token not \
TypeError: _tokenize() got an unexpected keyword argument 'pad_to_max_length'
```

A similar issue occurs if I remove the `pad_to_max_length` keyword; then `return_attention_masks` is the unexpected keyword.

## Expected behavior

Expected: the function returns without error a dict with the attention masks, padded sequence, and some other info, as specified by the documentation.

## Environment info

- `transformers` version: 2.1.1
- Platform: WSL 2, Ubuntu 18.04.4 LTS
- Python version: 3.8
- PyTorch version (GPU?): 1.4.0
- Tensorflow version (GPU?): N/A
- Using GPU in script?: no
- Using distributed or parallel set-up in script?: no


Same logits value for different input

Need more information

Further information is requested

# ❓ Questions & Help
## Details
I use BartForSequenceClassification pre-trained model from the HuggingFace Transformers library. During training phase logits from classification head gets different values, but during validation phase all logits values are equal even for different input texts.

I use BartTokenizer.batch_encode_plus to encode the text before feeding into the model.

I fine-tuned the model for 1 epoch using the following code:

```
config = BartConfig.from_pretrained(model_name)
config.num_labels = 3
config.output_hidden_states = False
config.output_attentions = False

transformer_model = BartForSequenceClassification.from_pretrained(model_name, config=config)
transformer_model.cuda();
optimizer = AdamW(transformer_model.parameters())

NUM_TRAIN_EPOCHS = 1
print("Training:")
for i in range(1, 1+NUM_TRAIN_EPOCHS):
  transformer_model.train()
  for batch_index, (batch_input_ids, batch_attention_masks, batch_y) in enumerate(train_dataloader, start=1):
    batch_input_ids_cuda = batch_input_ids.to(device)
    batch_attention_masks_cuda = batch_attention_masks.to(device)
    batch_y_cuda = batch_y.to(device)
    loss, logits, _ =  transformer_model(batch_input_ids_cuda, attention_mask=batch_attention_masks_cuda, labels=batch_y_cuda)
    transformer_model.zero_grad()
    loss.backward()
    optimizer.step()
```

And for validation I use the following code:

```
transformer_model.eval()
with torch.no_grad():
 for batch_index, (batch_input_ids, batch_attention_masks, batch_y) in enumerate(test_dataloader, start=1):
    batch_input_ids_cuda = batch_input_ids.to(device)
    batch_attention_masks_cuda = batch_attention_masks.to(device)
    batch_y_cuda = batch_y.to(device)
    loss, logits, _ =  transformer_model(batch_input_ids_cuda, attention_mask=batch_attention_masks_cuda, labels=batch_y_cuda)
    print("Input ids:", batch_input_ids_cuda)
    print("Attention masks:", batch_attention_masks_cuda)
    print("Logits:", logits)
```

Output of validation phase is:

```
Input ids: tensor([[    0,  3655,     9,  ...,     1,     1,     1],
        [    0, 31524,   347,  ...,     1,     1,     1],
        [    0, 12806, 24220,  ...,     1,     1,     1],
        ...,
        [    0,  8518,  7432,  ...,     1,     1,     1],
        [    0, 15006, 23613,  ...,     1,     1,     1],
        [    0, 14729, 13178,  ...,     1,     1,     1]], device='cuda:0')
Attention masks: tensor([[1, 1, 1,  ..., 0, 0, 0],
        [1, 1, 1,  ..., 0, 0, 0],
        [1, 1, 1,  ..., 0, 0, 0],
        ...,
        [1, 1, 1,  ..., 0, 0, 0],
        [1, 1, 1,  ..., 0, 0, 0],
        [1, 1, 1,  ..., 0, 0, 0]], device='cuda:0')
Logits: tensor([[-1.3014, -0.7394,  0.7862],
        [-1.3014, -0.7394,  0.7862],
        [-1.3014, -0.7394,  0.7862],
        [-1.3014, -0.7394,  0.7862],
        [-1.3014, -0.7394,  0.7862],
        [-1.3014, -0.7394,  0.7862],
        [-1.3014, -0.7394,  0.7862],
        [-1.3014, -0.7394,  0.7862],
        [-1.3014, -0.7394,  0.7862],
        [-1.3014, -0.7394,  0.7862],
        [-1.3014, -0.7394,  0.7862],
        [-1.3014, -0.7394,  0.7862],
        [-1.3014, -0.7394,  0.7862],
        [-1.3014, -0.7394,  0.7862],
        [-1.3014, -0.7394,  0.7862],
        [-1.3014, -0.7394,  0.7862]], device='cuda:0')
```

Why all the logits have exact same value for every input?

<!-- You should first ask your question on SO, and only if
     you didn't get an answer ask it here on GitHub. -->
**A link to original question on Stack Overflow**: https://stackoverflow.com/questions/62097267/same-logits-value-for-different-input-in-huggingfaces-transformer


Compressive Transformer

# 🌟 New model addition

## Model description

Title	Compressive Transformers for Long-Range Sequence Modelling (ICLR '20)

arXiv	1911.05507

Blog	A new model and dataset for long-range memory

__Compressive Transformer__ is an attentive sequence model which __compresses past memories__ for long-range sequence learning. The idea is similar to [Transformer-XL](https://arxiv.org/pdf/1901.02860.pdf). However, the memories are compressed in __Compressive Transformer__, making it leverage longer past memories compared to __Transformer-XL__.

## Open source status

- [ ] the model implementation is available
- [ ] the model weights are available
- [ ] who are the authors


Update HooshvareLab/bert-base-parsbert-uncased

model card

Related to pretrained model cards

mBERT results added regarding NER datasets!


[pipeline] Tokenizer should not add special tokens for text generation

This PR fixes generation in pipelines for all models whose tokenizer adds special tokens to the input, *e.g.* XLNet.

I think this is good for now, but I think the `_parse_and_tokenize()` function needs a larger refactoring to allow more flexibility in the future, also see Issue: #4501


AutoModel.from_config loads random parameter values.

# 🐛 Bug

## Information

Model I am using (Bert, XLNet ...): Bert

Language I am using the model on (English, Chinese ...): English

The problem arises when using:
* [ ] the official example scripts: (give details below)
* [x] my own modified scripts: (give details below)
Model parameters are (apparently) random initialized when using `AutoModel.from_config`.

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [x] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:

1. `git clone https://github.com/gkutiel/transformers-bug`
2. `cd transformers-bug`
3. `pipenv shell`
4. `pipenv install`
5. `python main.py`

```python
from transformers import (
    AutoModel,
    AutoConfig,
)

pretrained = 'bert-base-uncased'

model_from_pretrained = AutoModel.from_pretrained(pretrained)
model_from_config = AutoModel.from_config(AutoConfig.from_pretrained(pretrained))

model_from_pretrained_params = list(model_from_pretrained.parameters())
model_from_config_params = list(model_from_config.parameters())

assert len(model_from_pretrained_params) == len(model_from_config_params)

model_from_pretrained_first_param = model_from_pretrained_params[0][0][0]
model_from_config_first_param = model_from_config_params[0][0][0]

assert model_from_pretrained_first_param == model_from_config_first_param, (
    f'{model_from_pretrained_first_param} != {model_from_config_first_param}'
)
```
## Expected behavior
An assertion error should not happen.

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 2.10.0
- Platform: MacOS
- Python version:3.6
- PyTorch version (GPU?):
- Tensorflow version (GPU?):
- Using GPU in script?:
- Using distributed or parallel set-up in script?:


Create README.md

Related to pretrained model cards


when I encode [unused1], return not one token

Core: Tokenization

Internals of the library; Tokenization.

# 🐛 Bug

## Information

Model I am using (Bert, XLNet ...): bert

Language I am using the model on (English, Chinese ...): English

The problem arises when using: tokenizer.encode('[unused1]')
* [ ] the official example scripts: (give details below)
* [ ] my own modified scripts: (give details below)

The tasks I am working on is:relation extraction
* [ ] an official GLUE/SQUaD task: (give the name)
* [ ] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:

1. tokenizer.encode("[unused1]")
2. but return not one token, if using keras-bert, it will return me only one token
3.

<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

## Expected behavior

<!-- A clear and concise description of what you would expect to happen. -->

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: least version
- Platform:
- Python version: 3.7
- PyTorch version (GPU?):  1.1.0
- Tensorflow version (GPU?):
- Using GPU in script?:
- Using distributed or parallel set-up in script?:


XLNet Generation appears to reference padding text in run_generation script

Ex: Generation

Natural Language Generation

When generating with XLNet in the `run_generation.py` script, the outputs seem to reference the context from the padding text. For instance, given the prompt "We propose a", XLNet generates "We propose a boy Go Ya Ya, a young Iriel Farg, to be named Rasputin."

This seems to reference the padding text:
https://github.com/huggingface/transformers/blob/0866669e751bef636fa693b704a28c1fea9a17f3/examples/text-generation/run_generation.py#L62-L71

From what I understand, this padding text should not influence the generation, since the padding ends with an end of sentence token. Is this behavior expected?

Full command I used for reference:
```bash
python -m examples.run_generation --model_type xlnet --model_name_or_path xlnet-base-cased --prompt "We propose a"
```


NER: Add new WNUT’17 example

Hi,

this PR extends the NER example section, and adds an extra section for fine-tuning a NER model on the (difficult) WNUT’17 shared task:

> The WNUT’17 shared task focuses on identifying unusual, previously-unseen entities in the context of emerging discussions.
> Named entities form the basis of many modern approaches to other tasks (like event clustering and summarization), but recall on
> them is a real problem in noisy text - even among annotators. This drop tends to be due to novel entities and surface forms.

I also added my pre-processing script, that splits longer sentences into smaller ones (once the max. subtoken length is reached).


[EncoderDecoder] Fix initialization and save/load bug

This PR fixes two bugs:

- Cross attention layers were not initialized when initiating the `EncoderDecoderModel` via `from_encoder_decoder_pretrained()`. Thanks goes to https://github.com/huggingface/transformers/issues/4293 for finding this bug! A slow test is including in this PR to prevent future bugs of this  kind.

- Saving / loading of pretrained models didn't work because the weights were initialized from the `model_to_load` weights because of a missing `base_model_prefix` in the `EncoderDecoderModel` class. Another slow test is included in this PR to prevent future bugs. I think this problem was mentioned in this issue: https://github.com/huggingface/transformers/issues/4517. I didn't rerun the code attached here: https://github.com/huggingface/transformers/issues/4517#issuecomment-636189365 but I'm quite positive that this is the bug that will be fixed in this PR.

## IMPORTANT
*To everybody who has been training Bert2Bert using the EncoderDecoder framework: Training with the EncoderDecoderModel before this PR did not work because there were no cross attention layers to be trained if you initialized your `EncoderDecoderModel` using `.from_encoder_decoder_pretrained(...)`.  - I'm very sorry for the wasted compute and energy! Training should work now. I will add an encoder decoder notebook in the next 1-2 weeks showing in-detail how Bert2Bert can be used with `EncoderDecoder`*

This regards the Issues: #4445, #4647, #4517, #4443, #4293, #4640


GPT-3

Paper: https://arxiv.org/pdf/2005.14165

GitHub: https://github.com/openai/gpt-3

Author: @8enmann


Documentation for non-nlp experts

# 🚀 Feature request

<!-- A clear and concise description of the feature proposal.
     Please provide a link to the paper and code in case they exist. -->

## Motivation
I want to use these models as an end-user, without having to read academic papers describing them.

I have been following deep learning in the field of computer vision, had no idea that NLP had advanced SO much (for me, Word2vec is still state of the art).

At work I have large amounts of text: articles, transcripts from voice and chat, chatbot exchanges, etc. I would love to Try out the functionality provided by the contributors here, but the documentation seems to assume one already know which models to use.

Perhaps someone can write a “Modern NLP for technical managers” type post (or link to an existing one).

I’m excited by the immense amount of work done here, but will have to start all the way back with RNNs and work my way up to this stuff. Hoping to be able to use this stuff without that detour!

<!-- Please outline the motivation for the proposal. Is your feature request
     related to a problem? e.g., I'm always frustrated when [...]. If this is related
     to another GitHub issue, please link here too. -->

## Your contribution

<!-- Is there any way that you could help, e.g. by submitting a PR?
     Make sure to read the CONTRIBUTING.MD readme:
     https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md -->


Include `nlp` notebook for model evaluation


Gpt2 generation of text larger than 1024

Natural Language Generation

# ❓ Questions & Help

<!-- The GitHub issue tracker is primarly intended for bugs, feature requests,
     new models and benchmarks, and migration questions. For all other questions,
     we direct you to Stack Overflow (SO) where a whole community of PyTorch and
     Tensorflow enthusiast can help you out. Make sure to tag your question with the
     right deep learning framework as well as the huggingface-transformers tag:
     https://stackoverflow.com/questions/tagged/huggingface-transformers

     If your question wasn't answered after a period of time on Stack Overflow, you
     can always open a question on GitHub. You should then link to the SO question
     that you posted.
     -->

## Details
<!-- Description of your issue -->

I know the context supported by GPT2 is 1024, but I assume there's some technique they utilized to train and generate text longer than that in their results. Also, I saw many gpt2-based repos training text with length longer than 1024. But when I tried generating text longer than 1024 it throws a runtime error :The size of tensor a (1025) must match the size of tensor b (1024) at non-singleton dimension 3. I have the following questions:
1) Shouldn't it be possible to generate longer text since a sliding window is used?
2) Can you please explain what's necessary to generate longer text? What changes will I have to make to the run_generation.py code?

<!-- You should first ask your question on SO, and only if
     you didn't get an answer ask it here on GitHub. -->