noamaanMulla-03/transformers-issues huggingface dataset:
Expose all constructor parameters for BertTokenizerFast
Author: mfuntowicz
Pull request: yes

Signed-off-by: Morgan Funtowicz <morgan@huggingface.co>

Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/2921?src=pr&el=h1) Report
> Merging [#2921](https://codecov.io/gh/huggingface/transformers/pull/2921?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/d490b5d5003654f104af3abd0556e598335b5650?src=pr&el=desc) will **not change** coverage.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/2921/graphs/tree.svg?width=650&token=9qOlN6Hb1c&height=150&src=pr)](https://codecov.io/gh/huggingface/transformers/pull/2921?src=pr&el=tree)

```diff
@@           Coverage Diff           @@
##           master    #2921   +/-   ##
=======================================
  Coverage   75.35%   75.35%           
=======================================
  Files          94       94           
  Lines       15444    15444           
=======================================
  Hits        11638    11638           
  Misses       3806     3806
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/2921?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [src/transformers/tokenization\_bert.py](https://codecov.io/gh/huggingface/transformers/pull/2921/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy90b2tlbml6YXRpb25fYmVydC5weQ==) | `96.99% <ø> (ø)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/2921?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/2921?src=pr&el=footer). Last update [d490b5d...21ac4a0](https://codecov.io/gh/huggingface/transformers/pull/2921?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
Tokenizer fast warnings
Author: mfuntowicz
Pull request: yes

- Warning abut padding should not trigger so often now, especially when no padding strategy is provided by the user.

- RoberTa warning is now in RoberTaTokenizer, not GPT2 base class.

Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/2922?src=pr&el=h1) Report
> Merging [#2922](https://codecov.io/gh/huggingface/transformers/pull/2922?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/d490b5d5003654f104af3abd0556e598335b5650?src=pr&el=desc) will **decrease** coverage by `<.01%`.
> The diff coverage is `100%`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/2922/graphs/tree.svg?width=650&token=9qOlN6Hb1c&height=150&src=pr)](https://codecov.io/gh/huggingface/transformers/pull/2922?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master    #2922      +/-   ##
==========================================
- Coverage   75.35%   75.35%   -0.01%     
==========================================
  Files          94       94              
  Lines       15444    15445       +1     
==========================================
  Hits        11638    11638              
- Misses       3806     3807       +1
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/2922?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [src/transformers/tokenization\_gpt2.py](https://codecov.io/gh/huggingface/transformers/pull/2922/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy90b2tlbml6YXRpb25fZ3B0Mi5weQ==) | `96.82% <ø> (-0.03%)` | :arrow_down: |
| [src/transformers/tokenization\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/2922/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy90b2tlbml6YXRpb25fdXRpbHMucHk=) | `90.45% <100%> (-0.14%)` | :arrow_down: |
| [src/transformers/tokenization\_roberta.py](https://codecov.io/gh/huggingface/transformers/pull/2922/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy90b2tlbml6YXRpb25fcm9iZXJ0YS5weQ==) | `100% <100%> (ø)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/2922?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/2922?src=pr&el=footer). Last update [d490b5d...6a55286](https://codecov.io/gh/huggingface/transformers/pull/2922?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
Small fix: default args for torch-lightning
Author: srush
Pull request: yes

Fix to the default argument passing to torch-lightning.

Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/2937?src=pr&el=h1) Report
> Merging [#2937](https://codecov.io/gh/huggingface/transformers/pull/2937?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/e2a6445ebbc36121817c1f605d9a09a335f5fba5?src=pr&el=desc) will **decrease** coverage by `1.06%`.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/2937/graphs/tree.svg?width=650&token=9qOlN6Hb1c&height=150&src=pr)](https://codecov.io/gh/huggingface/transformers/pull/2937?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master    #2937      +/-   ##
==========================================
- Coverage   75.35%   74.28%   -1.07%     
==========================================
  Files          94       94              
  Lines       15445    15445              
==========================================
- Hits        11638    11473     -165     
- Misses       3807     3972     +165
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/2937?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [src/transformers/modeling\_tf\_pytorch\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/2937/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ190Zl9weXRvcmNoX3V0aWxzLnB5) | `8.72% <0%> (-81.21%)` | :arrow_down: |
| [src/transformers/modeling\_roberta.py](https://codecov.io/gh/huggingface/transformers/pull/2937/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ19yb2JlcnRhLnB5) | `85.91% <0%> (-9.86%)` | :arrow_down: |
| [src/transformers/modeling\_xlnet.py](https://codecov.io/gh/huggingface/transformers/pull/2937/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ194bG5ldC5weQ==) | `70.86% <0%> (-2.3%)` | :arrow_down: |
| [src/transformers/modeling\_ctrl.py](https://codecov.io/gh/huggingface/transformers/pull/2937/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ19jdHJsLnB5) | `92.07% <0%> (-2.21%)` | :arrow_down: |
| [src/transformers/modeling\_openai.py](https://codecov.io/gh/huggingface/transformers/pull/2937/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ19vcGVuYWkucHk=) | `80.2% <0%> (-1.35%)` | :arrow_down: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/2937?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/2937?src=pr&el=footer). Last update [e2a6445...34e9098](https://codecov.io/gh/huggingface/transformers/pull/2937?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
Fix InputExample docstring
Author: scottgigante
Pull request: yes

![image](https://user-images.githubusercontent.com/8499679/74761217-a1458700-5249-11ea-9fcf-0689c60ad5af.png)


Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/2891?src=pr&el=h1) Report
> Merging [#2891](https://codecov.io/gh/huggingface/transformers/pull/2891?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/2ae98336d17fceea7506af9880b862b6252a38f6?src=pr&el=desc) will **not change** coverage.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/2891/graphs/tree.svg?width=650&token=9qOlN6Hb1c&height=150&src=pr)](https://codecov.io/gh/huggingface/transformers/pull/2891?src=pr&el=tree)

```diff
@@           Coverage Diff           @@
##           master    #2891   +/-   ##
=======================================
  Coverage   75.06%   75.06%           
=======================================
  Files          94       94           
  Lines       15288    15288           
=======================================
  Hits        11476    11476           
  Misses       3812     3812
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/2891?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [src/transformers/data/processors/utils.py](https://codecov.io/gh/huggingface/transformers/pull/2891/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9kYXRhL3Byb2Nlc3NvcnMvdXRpbHMucHk=) | `21.73% <ø> (ø)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/2891?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/2891?src=pr&el=footer). Last update [2ae9833...e0b3974](https://codecov.io/gh/huggingface/transformers/pull/2891?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


Thanks!

Thank you! Easiest review ever! Keep em coming :)
Removed unused fields in DistilBert TransformerBlock
Author: guillaume-be
Pull request: yes

A few fields in the TransformerBlock are unused - this small PR cleans it up.


Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/2710?src=pr&el=h1) Report
> Merging [#2710](https://codecov.io/gh/huggingface/transformers/pull/2710?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/2ba147ecffa28e5a4f96eebd09dcd642117dedae?src=pr&el=desc) will **decrease** coverage by `0.27%`.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/2710/graphs/tree.svg?width=650&token=9qOlN6Hb1c&height=150&src=pr)](https://codecov.io/gh/huggingface/transformers/pull/2710?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master    #2710      +/-   ##
==========================================
- Coverage   74.09%   73.81%   -0.28%     
==========================================
  Files          93       93              
  Lines       15248    15243       -5     
==========================================
- Hits        11298    11252      -46     
- Misses       3950     3991      +41
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/2710?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [src/transformers/modeling\_distilbert.py](https://codecov.io/gh/huggingface/transformers/pull/2710/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ19kaXN0aWxiZXJ0LnB5) | `95.79% <ø> (-0.07%)` | :arrow_down: |
| [src/transformers/modeling\_tf\_auto.py](https://codecov.io/gh/huggingface/transformers/pull/2710/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ190Zl9hdXRvLnB5) | `52.94% <0%> (-21.57%)` | :arrow_down: |
| [src/transformers/pipelines.py](https://codecov.io/gh/huggingface/transformers/pull/2710/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9waXBlbGluZXMucHk=) | `68.79% <0%> (-3.33%)` | :arrow_down: |
| [src/transformers/tokenization\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/2710/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy90b2tlbml6YXRpb25fdXRpbHMucHk=) | `84.87% <0%> (-0.82%)` | :arrow_down: |
| [src/transformers/modeling\_tf\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/2710/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ190Zl91dGlscy5weQ==) | `92.3% <0%> (-0.52%)` | :arrow_down: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/2710?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/2710?src=pr&el=footer). Last update [2ba147e...d40db22](https://codecov.io/gh/huggingface/transformers/pull/2710?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
🌟 BART
Author: astariul
Pull request: no

# 🌟New model addition

## Model description

method for pre-training seq2seq models by de-noising text. BART outperforms previous work on a bunch of generation tasks (summarization/dialogue/QA), while getting similar performance to RoBERTa on SQuAD/GLUE

[BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://arxiv.org/pdf/1910.13461.pdf)

Amazing results on text summarization.

## Open Source status

* [x] the model implementation is available: not yet
* [x] the model weights are available: not yet
* [ ] who are the authors: @yinhanliu @ernamangoyal 

Comments:
@thomwolf another encoder-decoder

Was released today: https://github.com/pytorch/fairseq/tree/master/examples/bart 🎉

Let me know if you guys plan to add xsum/eli5/cnn-dm ft with our released bart into hugging face. 

Is there any news on this?

any progress on this one? also thanks :)

I'm getting started on this Feb 4!
RobertaTokenizer token type issue
Author: CZWin32768
Pull request: no
Labels: name: wontfix

## ❓ Questions & Help

<!-- A clear and concise description of the question. -->

Why the two middle `<\s>` are both assigned with token type 0?

https://github.com/huggingface/transformers/blob/e92bcb7eb6c5b9b6ed313cc74abaab50b3dc674f/transformers/tokenization_roberta.py#L149

Could this one be better?
```python
return len(cls + token_ids_0 + sep) * [0] + len(sep + token_ids_1 + sep) * [1] 
```

Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


having the same question... why add two seps instead of one?
Doubts on modeling_gpt2.py
Author: shashankMadan-designEsthetics
Pull request: no
Labels: name: wontfix

## ❓ Questions & Help

<!-- A clear and concise description of the question. -->
Hi, I've been going through the gpt2 source code and i was tracing out how the self attention and feed forward work basically we have `Block` which is a decoder consisting of other 2 segments `Attention` and `MLP`. I was also reading a blog where it mentions the `queries` has to be learned i finally saw the class responsible for that is `Conv1D`
```
class Conv1D(nn.Module):
    def __init__(self, nf, nx):
        """ Conv1D layer as defined by Radford et al. for OpenAI GPT (and also used in GPT-2)
            Basically works like a Linear layer but the weights are transposed
        """
        super(Conv1D, self).__init__()
        self.nf = nf
        w = torch.empty(nx, nf)
        nn.init.normal_(w, std=0.02)
        self.weight = nn.Parameter(w)
        self.bias = nn.Parameter(torch.zeros(nf))

    def forward(self, x):
        size_out = x.size()[:-1] + (self.nf,)
        x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight)
        x = x.view(*size_out)
        return x

```

Here we have a affine function `addmm` is how we do... but i was expecting `train optimer.step, loss` basically all the thing that goes into training in pytorch. Can anyone elaborate on it?

Then in `GPT2LMHeadModel` we use a linear layer to represent `vocab_embed` and we produce `logits` by multiplying it with the transformer output so if the linear layer is trained what is the use of `from_pretrained` anyway?

I am sure this may perhaps be silly questions but i'd like to get some help here.

Thanks a lot.

Comments:
Can anyone answer it please...

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
fp16 is not compatible with the current activation code when pytorch is less than 1.4.0
Author: bcmi220
Pull request: no

[gelu = getattr(F, "gelu", _gelu_python)](https://github.com/huggingface/transformers/blob/53ce3854a16ad2a715bc6ac8af3e30c18b5a1d11/src/transformers/activations.py#L21)

should be changed to:

```python
if torch.__version__ < '1.4.0':
    gelu = _gelu_python
else:
    gelu = getattr(F, "gelu", _gelu_python)
```



Comments:
This is a duplicate of https://github.com/huggingface/transformers/issues/2940
Finding the right keras loss and metric for SQuAD
Author: jwallat
Pull request: no

## ❓ Questions & Help

<!-- A clear and concise description of the question. -->
I am trying to build a simple example with BERT for QA (on SQuAD). The goal is getting it similarly as simple as [the GLUE example from the repository](https://github.com/huggingface/transformers#quick-tour-tf-20-training-and-pytorch-interoperability). 

The problem I am facing is finding the accurate loss function and metric. According to the [documentation](https://huggingface.co/transformers/model_doc/bert.html#tfbertforquestionanswering), TFBertForQuestionAnswering will return the logits for a span_start and span_end prediction. 
Since we want one single loss for both predictions, one could use the reduced sum of the both categorical crossentropies of the span predictions.

Is that a sensible way to do it? 
Is there another, better way? 
Can we "stack" losses in keras or is it just not possible? 

I am thankful for any help.

For reference: My current state can be found in [this colab notebook](https://colab.research.google.com/drive/1xDpV0z3432mnqdvDC40kMi-KQWxiKPQK)
 


Comments:
In case @jplu has an insight on this?

Hi @jwallat,

You might have two solutions to solve your issue:

* Implement your own loss function that you can give to the compile method (see the official Tensorflow [doc](https://www.tensorflow.org/api_docs/python/tf/keras/Model?version=stable#compile))
* Implement a custom training loop such as the `train` function in the [NER example](https://github.com/huggingface/transformers/blob/master/examples/run_tf_ner.py#L154)
"config.json" does not include correct "id2label" and "label2id" after finetuning on NER task
Author: lecidhugo
Pull request: no

## 🐛 Bug

<!-- Important information -->

Model I am using (Bert, XLNet....): xlmroberta

Language I am using the model on (English, Chinese....): English

The problem arise when using:
* [ ] the official example scripts: (give details)
I use script `run_ner.py` in order to finetune xlmroberta on conll03 dataset.

The script executed with no problems. But the file "config.json" in the output directory is not correct.


The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
Task: NER
Dataset:  conll03


## To Reproduce

Steps to reproduce the behavior:

1.
I run the script: run_ner.py as follows:
`python run_ner.py --data_dir 0-data --model_type 'xlmroberta' --model_name_or_path 'xlm-roberta-large' --output_dir 1-out --max_seq_length 32 --do_train --do_eval --per_gpu_train_batch_size 8 --no_cuda --evaluate_during_training --logging_steps 1756 --save_steps 1756 --eval_all_checkpoints`
2. Go to the output directory. The file "config.json"  contains :
"label2id": {
    "LABEL_0": 0,
    "LABEL_1": 1
  },
and
"id2label": {
    "0": "LABEL_0",
    "1": "LABEL_1"
  },
which are not expected in NER
3.

<!-- If you have a code sample, error messages, stack traces, please provide it here as well. -->

## Expected behavior

<!-- A clear and concise description of what you expected to happen. -->
I expect that "config.json" contains something like:
"id2label": {
    "0": "B-LOC",
    "1": "B-MISC",
    "2": "B-ORG",
    "3": "I-LOC",
    "4": "I-MISC",
    "5": "I-ORG",
    "6": "I-PER",
    "7": "O"
  },
and
"label2id": {
    "B-LOC": 0,
    "B-MISC": 1,
    "B-ORG": 2,
    "I-LOC": 3,
    "I-MISC": 4,
    "I-ORG": 5,
    "I-PER": 6,
    "O": 7
  },
## Environment

* OS: Ubuntu 18.04
* Python version: 3.6.9
* PyTorch version: 1.3.1
* PyTorch Transformers version (or branch): 2.3.0
* Using GPU ? No
* Distributed or parallel setup ? No
* Any other relevant information:

## Additional context

<!-- Add any other context about the problem here. -->


Comments:
I checked the codes days before and 'label2id' and 'id2label' seemed not used and didn't influence the code execution.
output padding different to zero in embedding layer
Author: akamnev
Pull request: no
Labels: name: Core: Modeling

name: Usage

# 🐛 Bug
On embedding layer, the token corresponding to the padding does not return 0.
## Information

Model I am using (Bert):

Language I am using the model on (English):

The problem arises when using:
* [ ] the official example scripts: (give details below)
* [x] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [x] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:

```python
import torch
from transformers.tokenization_bert import BertTokenizer
from transformers.modeling_bert import BertEmbeddings

tokenizer = BertTokenizer.from_pretrained('bert-base-cased')

class Config:
    vocab_size = tokenizer.vocab_size
    hidden_size = 768
    max_position_embeddings = 512
    type_vocab_size = 2
    hidden_dropout_prob = 0.1
    layer_norm_eps = 1e-12

max_length = 10
sentence = "I eat a green apple"
tokens = tokenizer.encode(sentence)
tokens += [tokenizer.pad_token_id] * (max_length - len(tokens))
print(tokens)

embedding = BertEmbeddings(Config)
input_ids = torch.tensor([tokens])
emb = embedding(input_ids)

print(emb[0][-1])  # the last step should return 0 tensor
```
## Expected behavior
I expect to get a zero output tensor

## Environment info     
- `transformers` version: 2.3.0
- Platform: Ubuntu
- Python version: 3.7
- PyTorch version (GPU?): 1.3.1 (CPU)
- Tensorflow version (GPU?): no
- Using GPU in script?: no
- Using distributed or parallel set-up in script?: no


Comments:
Why? Typically, padding tokens are ignored in the model by the use of an attention mask. I don't understand why you want to only get the output of the embedding layer. (Perhaps you are confused with the good ol' word2vec embedding models, but you cannot/should not extract features from the embedding layer in a LM.)
About QuestionAnswering on SQuAD2.0 Dataset
Author: WenTingTseng
Pull request: no
Labels: name: wontfix

## ❓ Questions & Help

<!-- A clear and concise description of the question. -->
Do anyone understand the paper " QuestionAnswering on SQuAD2.0 Dataset" in section 5.1. It says
 "As we increase the number of epochs in training, the performance of the answerable questions is improved while the performance for the non-answerable questions drop hugely"
<img width="393" alt="擷取" src="https://user-images.githubusercontent.com/32416416/70907720-88033d00-2044-11ea-9ab9-1f2437aefd11.PNG">
I want to know why cause this situation?
And it also says  One possible solution is that, the no-answer indicator is only from the ﬁrst [CLS] token, the value of attentions to the [CLS] token may be much weaker than the word-word attention. Hence the Transformer may focus less on the attention associated with the [CLS] token.
what does this mean about " the Transformer may focus less on the attention associated with the [CLS] token."
Can anyone help? Thans a lot


Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.