noamaanMulla-03/transformers-issues huggingface dataset:
Create camembert-base-README.md
Author: benjamin-mlr
Pull request: yes
Labels: name: model card

First version of our model_card for the original uploaded CamemBERT. 
@louismartin @pjox

Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/3263?src=pr&el=h1) Report
> Merging [#3263](https://codecov.io/gh/huggingface/transformers/pull/3263?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/afea70c01c7d2a844662a4d66b9f9d933cc6449c?src=pr&el=desc) will **increase** coverage by `0.11%`.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/3263/graphs/tree.svg?width=650&token=9qOlN6Hb1c&height=150&src=pr)](https://codecov.io/gh/huggingface/transformers/pull/3263?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master    #3263      +/-   ##
==========================================
+ Coverage   77.82%   77.93%   +0.11%     
==========================================
  Files          98       98              
  Lines       16666    16666              
==========================================
+ Hits        12970    12989      +19     
+ Misses       3696     3677      -19
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/3263?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [src/transformers/modeling\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/3263/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ191dGlscy5weQ==) | `93.84% <0%> (+0.13%)` | :arrow_up: |
| [src/transformers/modeling\_tf\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/3263/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ190Zl91dGlscy5weQ==) | `85.68% <0%> (+3.22%)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/3263?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/3263?src=pr&el=footer). Last update [afea70c...14537eb](https://codecov.io/gh/huggingface/transformers/pull/3263?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


Merged it (not sure it was ready yet, but feel free to update in another PR).

**[Model page](https://huggingface.co/camembert-base)**

Let us know if we can help in any way @benjamin-mlr @louismartin @pjox

You can also add 
```
---
language: french
---
```

on top of the README for the model to pop up when looking for FR models
NER pipeline missing start/end
Author: petulla
Pull request: no
Labels: name: wontfix

name: Core: Pipeline

## 🚀 Feature

2.3 is a great release! Really excited for pipelines.

The feature to add is the start/end positions of the entities. 

Additionally, the option to show the recognized entity rather than in subword form would be more user-friendly as an API.

## Motivation

[The release mentions including positions of the entities](https://github.com/huggingface/transformers/releases/tag/v2.3.0). The start/end positions are not in `transformers.py`

## Additional context
This is really exciting and motivated me to use your module. I hope to make a PR in the future.


Comments:
That would be a great feature, +1.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Train custom NER model with new Pipeline
Author: kormilitzin
Pull request: no
Labels: name: wontfix

name: Core: Pipeline

## 🚀 Feature

New `Pipelines` feature is great! I am wondering whether it will be possible to implement a pre-training on domain specific data (similar to ULMFiT approach, unsupervised encoder-decoder) and then train a custom NER model with annotated data (similar to spaCy)?

Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
GLUE Benchmark Hyperparameters
Author: shreydesai
Pull request: no
Labels: name: wontfix

## ❓ Questions & Help

In the `run_glue.py` script, are the hyperparameters for running BERT, RoBERTa, ALBERT, etc. the exact same? The documentation does not seem to outline separate hyperparameters, but the papers of each respective model show different hyperparameter ranges. I'm wondering if this was taken account when reporting the benchmark results.

Comments:
Hi, each result available on the [example page](https://huggingface.co/transformers/examples.html) shows the command that was used, displaying the hyper-parameters that are different from the defaults.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Why does the BertForQuestionAnswering sample code duplicate the [CLS] token?
Author: jswift24
Pull request: no
Labels: name: wontfix

## ❓ Questions & Help

<!-- A clear and concise description of the question. -->
The BertForQuestionAnswering sample code creates duplicate [CLS] tokens. Wondering why:

```
tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')
model = BertForQuestionAnswering.from_pretrained('bert-large-uncased-whole-word-masking-finetuned-squad')
question, text = "Who was Jim Henson?", "Jim Henson was a nice puppet"
input_text = "[CLS] " + question + " [SEP] " + text + " [SEP]"
input_ids = tokenizer.encode(input_text)
token_type_ids = [0 if i <= input_ids.index(102) else 1 for i in range(len(input_ids))]
start_scores, end_scores = model(torch.tensor([input_ids]), token_type_ids=torch.tensor([token_type_ids]))
all_tokens = tokenizer.convert_ids_to_tokens(input_ids)
print(' '.join(all_tokens[torch.argmax(start_scores) : torch.argmax(end_scores)+1]))
# a nice puppet
tokenizer.decode(input_ids)
#'[CLS] [CLS] who was jim henson? [SEP] jim henson was a nice puppet [SEP] [SEP]'  
```

If I remove the extra [CLS], the extraction doesn't work. It's exactly two tokens off:
```
input_ids = tokenizer.encode(input_text, add_special_tokens=False)
...rerun same code as above...
print(' '.join(all_tokens[torch.argmax(start_scores) : torch.argmax(end_scores)+1]))
# was a
```
What am I doing wrong? How can I get the extraction working without duplicate [CLS] tokens? (and duplicate final [SEP] tokens BTW).

The sample code comes right from the docs: https://huggingface.co/transformers/model_doc/bert.html#bertforquestionanswering

Comments:
Indeed, this is a mistake, thank you for raising an issue. It should have been fixed with 74755c89b92e0c0c027221c13fd034afed4d2136.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
GLUE benchmark score for XLNet_base_cased?
Author: CapGOGO
Pull request: no
Labels: name: wontfix

## ❓ Questions & Help

Can someone provide the GLUE benchmark scores for different GLUE tasks. Or a script for preforming predictions on the test files will be really helpful.


Comments:
Have you looked at the [run_glue.py](https://github.com/huggingface/transformers/blob/master/examples/run_glue.py) script ?

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Why albert has a print statement during forward?
Author: ChristofHenkel
Pull request: no
Labels: name: wontfix

## ❓ Questions & Help

both AlbertTransformer and AlbertLayerGroup have print statements in the forward method, which messes up logging / printing during training


Comments:
This was an issue with a previous version of transformers. Please upgrade it to a more recent version for the warning to go away. Thank you.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Model name 'distilbert-base-german-cased' was not found in model name list.
Author: woiza
Pull request: no

This code is ok:

`from transformers import AutoModel, AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("distilbert-base-german-cased")
model = AutoModel.from_pretrained("distilbert-base-german-cased")
`

But the following fails:

`from transformers import BertForSequenceClassification, AdamW, BertConfig

model = BertForSequenceClassification.from_pretrained(
    "distilbert-base-german-cased", 
    num_labels = 2,   
    output_attentions = False, 
    output_hidden_states = False,
)
`

`OSError: Model name 'distilbert-base-german-cased' was not found in model name list. We assumed 'https://s3.amazonaws.com/models.huggingface.co/bert/distilbert-base-german-cased/config.json' was a path, a model identifier, or url to a configuration file named config.json or a directory containing such a file but couldn't find any such file at this path or url.`

Comments:
Hi @woiza,

try to use: `DistilBertForSequenceClassification` instead of `BertForSequenceClassification` :)



@stefan-it 

that worked, thank you!
added pad_to_max_length option to batch_encode_plus
Author: ameasure
Pull request: yes
Labels: name: wontfix


Comments:
Thanks @ameasure, do you think you could run the quality tool as defined in the contributing guidelines for that test `check_code_quality` to pass?

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
AttributeError: 'Sst2Processor' object has no attribute 'tfds_map'
Author: abb4s
Pull request: no
Labels: name: wontfix

## 🐛 Bug

<!-- Important information -->

hey I just wanted to test BERT for sst2. I have just changed official example script to this: 

```
import tensorflow as tf
import tensorflow_datasets
from transformers import *

# Load dataset, tokenizer, model from pretrained model/vocabulary
tokenizer = BertTokenizer.from_pretrained('bert-base-cased')
model = TFBertForSequenceClassification.from_pretrained('bert-base-cased')
data = tensorflow_datasets.load('glue/sst2')

# Prepare dataset for GLUE as a tf.data.Dataset instance
train_dataset = glue_convert_examples_to_features(data['train'], tokenizer, max_length=128, task='sst-2')
valid_dataset = glue_convert_examples_to_features(data['validation'], tokenizer, max_length=128, task='sst-2')
train_dataset = train_dataset.shuffle(100).batch(32).repeat(2)
valid_dataset = valid_dataset.batch(64)

# Prepare training: Compile tf.keras model with optimizer, loss and learning rate schedule 
optimizer = tf.keras.optimizers.Adam(learning_rate=3e-5, epsilon=1e-08, clipnorm=1.0)
loss = tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True)
metric = tf.keras.metrics.SparseCategoricalAccuracy('accuracy')
model.compile(optimizer=optimizer, loss=loss, metrics=[metric])

# Train and evaluate using tf.keras.Model.fit()
history = model.fit(train_dataset, epochs=2, steps_per_epoch=115,
                    validation_data=valid_dataset, validation_steps=7)

# Load the TensorFlow model in PyTorch for inspection
model.save_pretrained('./save/')
pytorch_model = BertForSequenceClassification.from_pretrained('./save/', from_tf=True)

# Quickly test a few predictions - MRPC is a paraphrasing task, let's see if our model learned the task
sentence_0 = "I didn't think this was as absolutely horrible."
inputs_1 = tokenizer.encode_plus(sentence_0, add_special_tokens=True, return_tensors='pt')


pred_1 = pytorch_model(inputs_1['input_ids'], token_type_ids=inputs_1['token_type_ids'])[0].argmax().item()
print("sentence_1 is", pred_1)

```
I'm using  google colab with tensorflow2.0 . the error is :

AttributeError: 'Sst2Processor' object has no attribute 'tfds_map'


Comments:
@abb4s A workaround would be downgrading `transformers` to `2.2.0`. It worked for me that way.

Indeed, this was an error introduced by #1548. It was patched by 1efc208. Thank you for raising this issue! 

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
How to pretrain BERT whole word masking (wwm) model?
Author: mralexis1
Pull request: no
Labels: name: wontfix

## 🚀 Feature

Code to pretrain BERT whole word masking (wwm) model

## Motivation

WWM offers better performance, but the current codebase doesn't seem to support this feature.

## Additional context

Related i #1352 

Comments:
Any ideas on whether this will be included sooner or later?

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
add gpt2-xl for tf
Author: patrickvonplaten
Pull request: yes

TF GPT2-XL is now added to AWS and can be loaded via:

```
from transformers import TFGPT2LMHeadModel
model = TFGPT2LMHeadModel.from_pretrained('gpt2-xl')
```

Thanks @bkkaggle for pointing this out! 

Comments:
Good to merge for me. Tested whether model can generate text and everything seems fine.
@julien-c 

# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/3268?src=pr&el=h1) Report
> Merging [#3268](https://codecov.io/gh/huggingface/transformers/pull/3268?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/cc4c37952a961f2d13e83f3d5ba6dab811d0bbfd&el=desc) will **increase** coverage by `0.19%`.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/3268/graphs/tree.svg?width=650&height=150&src=pr&token=9qOlN6Hb1c)](https://codecov.io/gh/huggingface/transformers/pull/3268?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master    #3268      +/-   ##
==========================================
+ Coverage   77.82%   78.01%   +0.19%     
==========================================
  Files          98       98              
  Lines       16666    16666              
==========================================
+ Hits        12970    13002      +32     
+ Misses       3696     3664      -32     
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/3268?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [src/transformers/modeling\_tf\_gpt2.py](https://codecov.io/gh/huggingface/transformers/pull/3268/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ190Zl9ncHQyLnB5) | `96.16% <ø> (ø)` | |
| [src/transformers/modeling\_tf\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/3268/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ190Zl91dGlscy5weQ==) | `88.19% <0.00%> (+5.72%)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/3268?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/3268?src=pr&el=footer). Last update [cc4c379...501291a](https://codecov.io/gh/huggingface/transformers/pull/3268?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
Using Transformer Library for code prediction
Author: ghost
Pull request: no

Dear all,

I am new in exploring transformer library. I would like to use transformer models to train on my own text corpus (.txt) containing C++ source code tokens seperated with space characters. I would like to provide tokenized C++ source code files from multiple repositories in textual format (.txt) , and function should give me trained models with accuracy results, which I can use for code prediction latter on.

I have came accross with [Deep TabNine](https://tabnine.com/blog/deep/), which has used GPT2. But, I donot know about the following:

1. How could I train tranformer's library GPT2 model for C++ tokenized code? 
2. Can I use all transformer models such as BERT, GPT-2, RoBERTa, XLM, DistilBert, XLNet etc to train on my own C++ tokenized code? 
3. If no, which one can be used which one cannot and how? 
4. Is it advisable to use transformer pretrained models by fine tunning over my own textual corpus of C++ tokens? or should I build trained models from the scratch by using transformer library?


Please let me know about it.


Comments:
You should probably train a model from scratch.

Here a few links that are relevant:

- our blog post on [how to train a model from scratch](https://huggingface.co/blog/how-to-train) using `transformers` and `tokenizers`.
- specifically on the topic of code, we just uploaded [CodeBERTa](https://huggingface.co/huggingface/CodeBERTa-small-v1#codeberta), a model pretrained on the `CodeSearchNet` dataset from GitHub (+ fine-tuned to a classification task)

Let us know how it goes.
[BART] FP16 testing fixes
Author: sshleifer
Pull request: yes

closes #3249: fp16 forward pass failing when no `decoder_attention_mask` provided. Adds test coverage.

closes #3265: test_generate_fp16 was failing since #3140   (by sending proper kwargs to `BartForConditionalGenerate.generate`)



Comments:
I have a suspicion that this will also fix the GPU test runner.
Is there a way to reduce the vocabulary size?
Author: snaik2016
Pull request: no
Labels: name: wontfix

## ❓ Questions & Help

For fine tuning task is it possible to reduce the vocabulary size?
does simply editing the vocab & config  file work?

<!-- A clear and concise description of the question. -->


Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
ALBERT model does not work as expected
Author: anhnt1
Pull request: no
Labels: name: wontfix

## ❓ Questions & Help

<!-- A clear and concise description of the question. -->
Hi,
I am new to Transformers. I tried the example for class AlbertForQuestionAnswering from huggingface.co. The results in each run are different and not correct. Please help.
Thanks,
Tuan Anh

Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Padding part output in BERT NER task is not [PAD]?
Author: DrDavidS
Pull request: no
Labels: name: wontfix

## ❓ Questions & Help

Hello ,my friends.

I have a problem while I am trying to do a Chinese NER task.

For easy to understand I will use some English words instead.

Assume padding length =128, here is the sentence:

> [CLS] Marilyn Monroe is an famous actress. [SEP] [PAD]  [PAD]  ...  [PAD] 

After I put it into `BertForTokenClassification` , I got a output like:

>[CLS] Marilyn Monroe is an famous actress.
>[CLS] B-PER I-PER O O O O O

That looks good ,but ,when output reached padding area it become strange:

>[SEP] [PAD] [PAD] [PAD] [PAD] [PAD] ...  [PAD]
>O O O O O  [CLS] [CLS] O ...O

it seems padding area including [SEP] token output randomly (they should be [SEP] and [PAD]), and most of them are 'O' with a littel [CLS] or [B-PER]\[I-PER].

I am confused about that.

I am sure taht:

- I already set `attention_masks` on padding area while training, attention_masks on [PAD] are 0.

- I already set `token_type_ids`, token_type_ids on [PAD] are 1.

and also in eval with `attention_mask` and `token_type_ids`.

So that's what I am depressed about, I have to cut output padding area before I solve this problem, I don't think it's a good idea.

Can someone help me? :(

Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
run_squad_w_distillation update
Author: simonepreite
Pull request: no
Labels: name: wontfix

## 🐛 Bug

<!-- Important information -->

Model I am using (Bert, XLNet....): DistilBERT and BERT

Language I am using the model on (English, Chinese....): English

The problem arise when using:
* [ ] the official example scripts: run_squad_w_distillation.py

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: SQuAD

## To Reproduce

Steps to reproduce the behavior:

1. Fine tuning on question answering from BERT to DistilBERT

<!-- If you have a code sample, error messages, stack traces, please provide it here as well. -->

## Expected behavior
The files  utils_squad and utils_squad_evaluate are missing right now, so to make this script working I had to restore the files from previous version of this repo. What I expected is to use the script without using tricks

## Environment

* OS: Ubuntu 16.04
* Python version: 3.5.6
* PyTorch version: 1.3.1
* Using GPU nvidia P100 (2x) or 1080Ti

## Additional context

<!-- Add any other context about the problem here. -->


Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
T5_INPUTS_DOCSTRING correct!?
Author: AndreSoble
Pull request: no
Labels: name: wontfix

Is this docstring even correct?

The [CLS] and [SEP] Tokens do not appear in the from google provided dictionaries for the pretrained "t5-base" model. If this is true could you then please provide a correct example of how to use the text generation feature of this almighty transformer? (BoolQ or QA) That would help me a lot.

Thanks!

```
T5_INPUTS_DOCSTRING = r"""
    Inputs:
        **input_ids**: ``torch.LongTensor`` of shape ``(batch_size, sequence_length)``:
            Indices of input sequence tokens in the vocabulary.
            To match pre-training, T5 input sequence should be formatted with [CLS] and [SEP] tokens as follows:
            (a) For sequence pairs:
                ``tokens:         [CLS] is this jack ##son ##ville ? [SEP] no it is not . [SEP]``
            (b) For single sequences:
                ``tokens:         [CLS] the dog is hairy . [SEP]``
            T5 is a model with relative position embeddings so you should be able to pad the inputs on
            the right or the left.
            Indices can be obtained using :class:`transformers.T5Tokenizer`.
            See :func:`transformers.PreTrainedTokenizer.encode` and
            :func:`transformers.PreTrainedTokenizer.convert_tokens_to_ids` for details.
        **attention_mask**: (`optional`) ``torch.FloatTensor`` of shape ``(batch_size, sequence_length)``:
            Mask to avoid performing attention on padding token indices.
            Mask values selected in ``[0, 1]``:
            ``1`` for tokens that are NOT MASKED, ``0`` for MASKED tokens.
        **head_mask**: (`optional`) ``torch.FloatTensor`` of shape ``(num_heads,)`` or ``(num_layers, num_heads)``:
            Mask to nullify selected heads of the self-attention modules.
            Mask values selected in ``[0, 1]``:
            ``1`` indicates the head is **not masked**, ``0`` indicates the head is **masked**.
"""

```

Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.