noamaanMulla-03/transformers-issues huggingface dataset:
What should I do if I want to change the padding idx in pytorch bert(Huggingface)?
Author: R-craft
Pull request: no


I find ,in RobertaModel class, the padding idx is limited to 1(is it true?),which is different from my tokenizer and data.So what should I do to change or just apply the model structure...

###
class RobertaEmbeddings(BertEmbeddings):
    """
    Same as BertEmbeddings with a tiny tweak for positional embeddings indexing.
    """

    def __init__(self, config):
        super().__init__(config)
        self.padding_idx = 1
        self.word_embeddings = nn.Embedding(config.vocab_size, config.hidden_size, padding_idx=self.padding_idx)
        self.position_embeddings = nn.Embedding(
            config.max_position_embeddings, config.hidden_size, padding_idx=self.padding_idx
        )

    def forward(self, input_ids=None, token_type_ids=None, position_ids=None, inputs_embeds=None):
        if position_ids is None:
            if input_ids is not None:
                # Create the position ids from the input token ids. Any padded tokens remain padded.
                position_ids = create_position_ids_from_input_ids(input_ids, self.padding_idx).to(input_ids.device)
            else:
                position_ids = self.create_position_ids_from_inputs_embeds(inputs_embeds)

        return super().forward(
            input_ids, token_type_ids=token_type_ids, position_ids=position_ids, inputs_embeds=inputs_embeds
        )

Comments:
Hi @R-craft, 

That's actually a bug. Thanks for spotting it :-) Will open a PR.
Bug in variable name in NER
Author: TarasPriadka
Pull request: no

# 🐛 Bug

## Information

Model I am using (Bert, XLNet ...): Irrelevant

Language I am using the model on (English, Chinese ...): Irrelevant

The problem arises when using:
* [x] the official example scripts: (give details below)
* [ ] my own modified scripts: (give details below)

The tasks I am working on is:
* [x] an official GLUE/SQUaD task: NER
* [ ] my own task or dataset: (give details below)

## Error

```
File "/transformers/examples/ner/utils_ner.py", line 123, in convert_examples_to_features special_tokens_count = tokenizer.num_added_tokens() 
AttributeError: 'BertTokenizer' object has no attribute 'num_added_tokens'
```

## Issue

After update to new Tokenizers, some util files are broken. Found one in examples/ner/utils_ner.py.
Need to change line 123 from num_added_tokens to num_special_tokens_to_add.


Comments:
I also see the same issue, has there been a fix for this?
AttributeError: 'BertTokenizer' object has no attribute 'num_added_tokens'

@minhtuev you can change `num_added_tokens` to `num_special_tokens_to_add`. This made the fix for me
How do I convert T5 checkpoint to pytorch bin to utilize fine-tuned model from finetune.py?
Author: enzoampil
Pull request: no

# ❓ Questions & Help

I've recently tried to fine-tune T5 using bart's [finetune.py](https://github.com/huggingface/transformers/blob/master/examples/summarization/bart/finetune.py) (supports T5 fine-tuning as well). The model is saved in a `ckpt` format when I wanted it as a pytorch `bin`, so I found this [page](https://huggingface.co/transformers/converting_tensorflow_models.html) in the docs on how to convert `ckpt` to `bin`. However, I don't see the T5 model in this page. 

Is it possible to use the CLI to convert T5 models?

I also have questions around fine-tuning T5 but will discuss details in a separate issue.

Comments:
Sorry to answer this late - can you try the following:
```python
model.from_pretrained(<path_to_model.ckpt>, from_tf=True) #even though you don't use TF
model.save_pretrained("./") # should be in .bin format
```

If this does not work try the following:
```python 
parser = argparse.ArgumentParser()
add_generic_args(parser, os.getcwd())
parser = SummarizationTrainer.add_model_specific_args(parser, os.getcwd())
args = parser.parse_args()
model = SummarizationTrainer(args)
model = model.load_from_checkpoint(<path_to_your_checkpoint>)
model.model.save_pretrained("./") # should be in bin format
```

Please do let me know if this works.

Actually someone else found a much better solution than I did :-) 
See: https://github.com/huggingface/transformers/issues/4144

Awesome this worked! Thanks so much @patrickvonplaten
problem about change from pytorch-pretrained-bert to transformers
Author: BiggHeadd
Pull request: no

I am doing the sentiment analysis with pytorch-pretrained-bert, it work correct, but when I change to transformers, the accuracy on the dev data is poor (but the accuracy with pytorch-pretrained-bert is quite higher). Besides, I notice the Migrating from pytorch-pretrained-bert to transformers module.
the input is:
[CLS] sentence [SEP] aspect [SEP]
and just using BertForSequenceClassification (also try add an classifier after BertModel), but all fail on transformers.
Can Sombody figure out what's wrong with it? Thanks!

Comments:
OK, the problem is, in transformers, forward() method args position is little bit different compare to pytorch-pretrained-bert. the order change from (token, segment, mask) to (token, mask, segment). So pass the args by name will always right, lol
Not able to import certain packages
Author: prajjwal1
Pull request: no

# 🐛 Bug

## Information
- I am using `run_glue.py`.
- Working with GLUE


## To reproduce

Steps to reproduce the behavior:
Even though these functions are imported in `__init__.py` (I have thoroughly checked myself), I'm not able to import `EvalPrediction`, `HfArgumentParser`, `Trainer`, `TrainingArguments`  


## Expected behavior
Should be able to import them. I'm sharing this [colab notebook](https://colab.research.google.com/drive/1CqWzt9A96iUT2vh47qwnMSDMsQHQkRSx) link.

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->
     
- `transformers` version: 2.8
- Platform: Ubuntu 18.04
- Python version: 1.5
- PyTorch version (GPU?): 1.5
- Tensorflow version (GPU?): NA
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: No


Comments:
You need to make sure you install from source as documented in the README
Fix markdown to show the results table properly
Author: mrm8488
Pull request: yes
Labels: name: model card


Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/4119?src=pr&el=h1) Report
> Merging [#4119](https://codecov.io/gh/huggingface/transformers/pull/4119?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/1cdd2ad2afb73f6af185aafecb7dd7941a90c4d1&el=desc) will **decrease** coverage by `0.01%`.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/4119/graphs/tree.svg?width=650&height=150&src=pr&token=9qOlN6Hb1c)](https://codecov.io/gh/huggingface/transformers/pull/4119?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master    #4119      +/-   ##
==========================================
- Coverage   78.85%   78.84%   -0.02%     
==========================================
  Files         114      114              
  Lines       18688    18688              
==========================================
- Hits        14737    14734       -3     
- Misses       3951     3954       +3     
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/4119?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [src/transformers/file\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/4119/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9maWxlX3V0aWxzLnB5) | `73.14% <0.00%> (-0.42%)` | :arrow_down: |
| [src/transformers/modeling\_tf\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/4119/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ190Zl91dGlscy5weQ==) | `92.61% <0.00%> (-0.17%)` | :arrow_down: |
| [src/transformers/modeling\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/4119/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ191dGlscy5weQ==) | `90.92% <0.00%> (-0.13%)` | :arrow_down: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/4119?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/4119?src=pr&el=footer). Last update [1cdd2ad...397a837](https://codecov.io/gh/huggingface/transformers/pull/4119?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
Fix overwrite_cache behaviour for pytorch lightning examples
Author: simonepri
Pull request: yes

cc: @nateraw

Ref: #3290

Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/4093?src=pr&el=h1) Report
> Merging [#4093](https://codecov.io/gh/huggingface/transformers/pull/4093?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/b8686174be75220d2c26a961597a39ef4921b616&el=desc) will **increase** coverage by `0.00%`.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/4093/graphs/tree.svg?width=650&height=150&src=pr&token=9qOlN6Hb1c)](https://codecov.io/gh/huggingface/transformers/pull/4093?src=pr&el=tree)

```diff
@@           Coverage Diff           @@
##           master    #4093   +/-   ##
=======================================
  Coverage   78.84%   78.85%           
=======================================
  Files         114      114           
  Lines       18691    18691           
=======================================
+ Hits        14737    14738    +1     
+ Misses       3954     3953    -1     
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/4093?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [src/transformers/trainer.py](https://codecov.io/gh/huggingface/transformers/pull/4093/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy90cmFpbmVyLnB5) | `43.90% <0.00%> (+0.34%)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/4093?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/4093?src=pr&el=footer). Last update [b868617...d0f7228](https://codecov.io/gh/huggingface/transformers/pull/4093?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


Not sure when it got broken, but this looks ok at first glance to me. I'll pull it down tonight as a sanity check to run it. 

Thank you! 

LGTM too 😄 Thank you again, @simonepri
[WIP] Add patience argument to run_language_modeling script
Author: thesamuel
Pull request: yes

# Summary
Often, we want to stop training if loss does not improve for a number of epochs. This PR adds a "patience" argument, which is a limit on the number of times we can get a non-improving eval loss before stopping training early.

It is implemented by other NLP frameworks, such as AllenNLP (see [trainer.py](https://github.com/allenai/allennlp/blob/master/allennlp/training/trainer.py#L95) and [metric_tracker.py](https://github.com/allenai/allennlp/blob/1a8a12cd1b065d74fec3d2e80105a684736ff709/allennlp/training/metric_tracker.py#L6)).

# Motivation
This feature allows faster fine-tuning by breaking the training loop early and avoids users the toil of checking metrics on Tensorboard.

# Caveats
Often, models are evaluated once per epoch, but run_lm_finetuning.py has an option to evaluate after a set number of model update steps (dictated by `--logging_steps` if `--evaluate_during_training` is true). Because of this, I've elected to tie patience to the number of evaluations without improvement in loss.

# To-do
- Add tests
- Fix long lines

Comments:
Sounds great! I'll go ahead and fix the code quality check.

Since `run_langauge_modeling.py` now uses the `Trainer` class, I'll likely create a new PR that adds patience to `Trainer`.
A small model for CTRL 
Author: D-i-l-r-u-k-s-h-i
Pull request: no
Labels: name: wontfix

name: Distillation

# 🚀 Feature request

A smaller version of the pre-trained model CTRL, related to the stack overflow question;
[https://stackoverflow.com/questions/60142937/huggingface-transformers-for-text-generation-with-ctrl]

## Motivation

I've been trying to generate text using CTRL and I run to a memory insufficiency, since it is a large model. And was wondering whether there would be a small version of CTRL like a Distill version like some of the other transformer models do.


Comments:
cc'ing @keskarnitish on this issue just in case!

Thank you, @julien-c.

hi, any update on this?

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Finetuning before feature extraction
Author: Stuffooh
Pull request: no

Hi,

Currently I am using the pipeline feature extraction to extract features with my own dataset as input. I was wondering if it is possible to finetune different models using my own dataset before using the pipeline for feature extraction and if so what will be the easiest way to do so?

In the past there used to be a lm_finetuning script example but this one is no longer available. I cannot find any examples or guides how to finetune different models on a personal dataset.

tl;dr is it possible to finetune different models on a personal dataset in just a few lines of code and if so how?

Thanks in advance,

A clueless person trying to learn more about the world of NLP.



Comments:
The script has been renamed [`run_language_modeling.py`](https://github.com/huggingface/transformers/blob/master/examples/run_language_modeling.py) to better reflect the fact that it can also be used to train a new model from scratch.

Let us know if it works.

Hi @julien-c  thanks for the quick reply!

That makes sense, I must have missed the fact it got renamed. I have one more question about the "new" script:

How difficult is it make to script compatible for finetuning a  new model like BART? Is it as simple as adding the model to the list inside the script or will it need a lot of workarounds to get it working?
[Reformer] Fix example and error message
Author: patrickvonplaten
Pull request: yes

Fix typos and improve error message.

Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/4191?src=pr&el=h1) Report
> Merging [#4191](https://codecov.io/gh/huggingface/transformers/pull/4191?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/96c78396ce1baf5e19c0618689005f93c7f42d79&el=desc) will **decrease** coverage by `0.00%`.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/4191/graphs/tree.svg?width=650&height=150&src=pr&token=9qOlN6Hb1c)](https://codecov.io/gh/huggingface/transformers/pull/4191?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master    #4191      +/-   ##
==========================================
- Coverage   78.41%   78.41%   -0.01%     
==========================================
  Files         120      120              
  Lines       19785    19785              
==========================================
- Hits        15515    15514       -1     
- Misses       4270     4271       +1     
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/4191?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [src/transformers/modeling\_reformer.py](https://codecov.io/gh/huggingface/transformers/pull/4191/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ19yZWZvcm1lci5weQ==) | `88.17% <ø> (ø)` | |
| [src/transformers/modeling\_tf\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/4191/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ190Zl91dGlscy5weQ==) | `92.77% <0.00%> (-0.17%)` | :arrow_down: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/4191?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/4191?src=pr&el=footer). Last update [96c7839...488d787](https://codecov.io/gh/huggingface/transformers/pull/4191?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
Paragraph re-ranking using MS MARCO dataset
Author: Sharathmk99
Pull request: no
Labels: name: wontfix

name: Usage

How can I use transformers package to train on MS MARCO dataset  by adding my own domain data. Or can I use pre trained model to add my own domain data?

SO link: https://stackoverflow.com/questions/60424723/paragraph-re-ranking-using-ms-marco-dataset

Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Language model training with NSP using run_language_modeling.py
Author: Stuffooh
Pull request: no
Labels: name: wontfix

When using "run_language_modeling.py" for fine-tuning BERT is only the MLM task used or also the NSP task as described in the paper?

Comments:
This is currently not supported. There used to be some community maintained examples IIRC. You can have a look here: https://github.com/huggingface/transformers/tree/1.1.0/examples/lm_finetuning
[qol] example scripts: parse args from .args file or JSON
Author: julien-c
Pull request: yes

You can either:
- pass the path to a json file as the unique argument 
- automatically load args from a .args file that's a sibling of the entrypoint script.

```json
{
  "model_name_or_path": "bert-base-cased",
  "task_name": "mnli",
  "data_dir": "./data/glue/MNLI",
  "output_dir": "./models/mnli"
}
```

```
--model_name_or_path distilroberta-base
--task_name mnli
```

I use the .args method myself, but I think both are ok.

Tagging @jplu and @stefan-it who have expressed interest in this.

Comments:
Thanks for adding this! Could you also integrate it into the `run_ner` example 🤔

Yes @stefan-it I'll add it to all (Trainer-updated) scripts.

Do you have a preference between the JSON approach and the args one? Or should we do both?

Perfect!!! it works like a charm. I think both would be nice.

Both would be nice, thanks!

Very nice!

@julien-c Would it be helpful when I open a PR that adds this to the `run_ner` example, or do you already have this feature on a branch 🤔

I do not have a branch that does that currently so feel free to do it (+ also run_language_modeling and run_multiple_choice if you're so inclined :)

@julien-c I am a bit late to the party, but I went over it quickly nevertheless. As my comments show, if the only function of `trim_suffix` is to remove the extension or replace it, it might be more readable to simply use the built-in `Path.with_extension` functionality.

If you agree I can do a PR for this.

Apart from that: great stuff. especially with a growing number of parameters and options, a config file is really welcome!

@BramVanroy Yes, I didn't know this feature, PR is welcome.
Rewritten batch support in pipelines.
Author: mfuntowicz
Pull request: yes

Batch support in Pipeline was confusing and not well tested. 

This PR rewrites all the content of `DefaultArgumentHandler` which handles most of the input conversions (args, kwargs, batched, etc.) and brings unit tests on this specific class.

Signed-off-by: Morgan Funtowicz <morgan@huggingface.co>

Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/4154?src=pr&el=h1) Report
> Merging [#4154](https://codecov.io/gh/huggingface/transformers/pull/4154?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/818463ee8eaf3a1cd5ddc2623789cbd7bb517d02&el=desc) will **increase** coverage by `0.03%`.
> The diff coverage is `96.42%`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/4154/graphs/tree.svg?width=650&height=150&src=pr&token=9qOlN6Hb1c)](https://codecov.io/gh/huggingface/transformers/pull/4154?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master    #4154      +/-   ##
==========================================
+ Coverage   78.79%   78.83%   +0.03%     
==========================================
  Files         114      114              
  Lines       18711    18726      +15     
==========================================
+ Hits        14743    14762      +19     
+ Misses       3968     3964       -4     
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/4154?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [src/transformers/pipelines.py](https://codecov.io/gh/huggingface/transformers/pull/4154/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9waXBlbGluZXMucHk=) | `76.27% <96.42%> (+1.32%)` | :arrow_up: |
| [src/transformers/modeling\_tf\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/4154/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ190Zl91dGlscy5weQ==) | `92.77% <0.00%> (+0.16%)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/4154?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/4154?src=pr&el=footer). Last update [818463e...95a257c](https://codecov.io/gh/huggingface/transformers/pull/4154?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


Ok so this should be ready to merge, right @mfuntowicz?

+1 Would love a merge here so I can more easily improve `TranslationPipeline`