noamaanMulla-03/transformers-issues huggingface dataset:
update .gitignore to ignore .swp files created when using vim
Author: patrickvonplaten
Pull request: yes

adds one line to .gitignore

Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/2881?src=pr&el=h1) Report
> Merging [#2881](https://codecov.io/gh/huggingface/transformers/pull/2881?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/6083c1566e261668a5de73cfe484c171ce232812?src=pr&el=desc) will **decrease** coverage by `1.07%`.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/2881/graphs/tree.svg?width=650&token=9qOlN6Hb1c&height=150&src=pr)](https://codecov.io/gh/huggingface/transformers/pull/2881?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master    #2881      +/-   ##
==========================================
- Coverage   75.06%   73.98%   -1.08%     
==========================================
  Files          94       94              
  Lines       15288    15288              
==========================================
- Hits        11476    11311     -165     
- Misses       3812     3977     +165
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/2881?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [src/transformers/modeling\_tf\_pytorch\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/2881/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ190Zl9weXRvcmNoX3V0aWxzLnB5) | `8.72% <0%> (-81.21%)` | :arrow_down: |
| [src/transformers/modeling\_roberta.py](https://codecov.io/gh/huggingface/transformers/pull/2881/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ19yb2JlcnRhLnB5) | `85.91% <0%> (-9.86%)` | :arrow_down: |
| [src/transformers/modeling\_xlnet.py](https://codecov.io/gh/huggingface/transformers/pull/2881/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ194bG5ldC5weQ==) | `70.86% <0%> (-2.3%)` | :arrow_down: |
| [src/transformers/modeling\_ctrl.py](https://codecov.io/gh/huggingface/transformers/pull/2881/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ19jdHJsLnB5) | `92.07% <0%> (-2.21%)` | :arrow_down: |
| [src/transformers/modeling\_openai.py](https://codecov.io/gh/huggingface/transformers/pull/2881/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ19vcGVuYWkucHk=) | `80.2% <0%> (-1.35%)` | :arrow_down: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/2881?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/2881?src=pr&el=footer). Last update [6083c15...fb4d8d0](https://codecov.io/gh/huggingface/transformers/pull/2881?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


Welcome @patrickvonplaten :)
ALBERT pretrained models uses wrong type of GELU activation
Author: Rexhaif
Pull request: no

## 🐛 Bug

<!-- Important information -->

Model I am using (Bert, XLNet....): ALBERT

Language I am using the model on (English, Chinese....): English

The problem arise when using:
* [x] my own modified scripts: i'm fine-tuning albert for multilabel-classifcation problem and then converting model into tf-lite format

The tasks I am working on is:
* [x] my own task or dataset: multilable text classification on SemEval 2018 task 1:E-c

## To Reproduce

Steps to reproduce the behavior:

1. Open any link to pretrained configuration at [transformers/configuration_albert.py](https://github.com/huggingface/transformers/blob/master/src/transformers/configuration_albert.py). For example: [albert-base-v2-config.json](https://s3.amazonaws.com/models.huggingface.co/bert/albert-base-v2-config.json). But problem applies to all the pretrained configs.
2. Check the value of property "hidden_act" (it will be gelu)
3. Realize, that gelu stands for bert-like implementation(see code [here](https://github.com/huggingface/transformers/blob/1ae63e4e097fe26c900783dd5c1710cf562e222e/src/transformers/modeling_bert.py#L152)), while original [code](https://github.com/google-research/ALBERT/blob/e350db671ae96b8345cd2c0ee1306713642b9810/modeling.py#L296) uses OpenAI-GPT - like gelu (it is defined in transformers as "gelu_new")

<!-- If you have a code sample, error messages, stack traces, please provide it here as well. -->

## Expected behavior

All the configuration files should contain "gelu_new" under "hidden_act" key.

## Environment

* OS: doesn't matter
* Python version: 3.7
* PyTorch version: 1.3.0
* PyTorch Transformers version (or branch): master
* Using GPU: doesn't matter
* Distributed of parallel setup: doesn't matter
* Any other relevant information:

## Additional context

<!-- Add any other context about the problem here. -->
While it possibly doesn't significantly change the model performance, it makes conferting model into tf-lite format a lot more tricky, because BERT-like gelu implementation uses tf.math.erf function, which is not in a tflite-builtins set. So i have to use tflite-select ops, which leads to using more heavy tflite-interpreter on android side. Also, it makes impossible to check the converted model performance on python side because default tensorflow-lite python interpreter can't interpret select-ops models.

Comments:
If you just change your config.json's `hidden_act` value locally you should still be able to load the pretrained weights and be able to convert the model to TFLite, right? 

Yes. Another option (my current workaround) is to explicitly specify hidden_act when creating model instance (via .from_pretrained(...)) during fine-tuning stage. 

It looks like you have fixed all ALBERT config files. Thanks!)
How  to run TFBERT model in disable_eager_execution() mode 
Author: JKP0
Pull request: no

How  to run `TFBERT `model in `disable_eager_execution()` mode.

If it is possible please let me know, thanks! 

Comments:
TFBERT model can only be loaded in tf>2.0 version.And,tf 2.0 onwards,eager_execution() is on by default
Supoort loading model weights from a single file.
Author: ljch2018
Pull request: no
Labels: name: wontfix

So far, transformers package supports loading a model from a directory, such as
```python
model = BertModel.from_pretrained('./test/saved_model/')  # E.g. model was saved using save_pretrained('./test/saved_model/')
```

Actually, it loads model weight from $directory/pytorch_model.bin.
Is it possible to load model weights from a file? 
such as 
```python
 tokenizer = BertTokenizer.from_pretrained('./test/saved_model/my_vocab.txt')
```

Because my network is too slow to download models from amazon server.
I have to download the model files manually and put them to transformers.
It is more convenient to load the downloaded model weights from a file than a directory.
```python
model = BertModel.from_pretrained('./test/model_zoo/bert-base-multilingual-cased-pytorch_model.bin')  
```
Thank you.

Comments:
As my best knowledge, you **can't** load a model _directly_ from a file. As said in #2094 by @LysandreJik, if you saved using the `save_pretrained` method, then the directory already should have a `config.json` file specifying the shape of the model, so you can simply load it using:
```
>>> import transformers
>>> from transformers import BertModel
>>> model = BertModel.from_pretrained("./test/saved_model/")
```

> So far, transformers package supports loading a model from a directory, such as
> 
> ```python
> model = BertModel.from_pretrained('./test/saved_model/')  # E.g. model was saved using save_pretrained('./test/saved_model/')
> ```
> 
> Actually, it loads model weight from $directory/pytorch_model.bin.
> Is it possible to load model weights from a file?
> such as
> 
> ```python
>  tokenizer = BertTokenizer.from_pretrained('./test/saved_model/my_vocab.txt')
> ```
> 
> Because my network is too slow to download models from amazon server.
> I have to download the model files manually and put them to transformers.
> It is more convenient to load the downloaded model weights from a file than a directory.
> 
> ```python
> model = BertModel.from_pretrained('./test/model_zoo/bert-base-multilingual-cased-pytorch_model.bin')  
> ```
> 
> Thank you.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Proposal: Update examples to utilize a new format.
Author: srush
Pull request: yes

This PR creates a new example coding style for the pytorch code. 

* Uses pytorch-lightning for the underlying training. 
* Separates out the base transformer loading from the individual training. 
* Moves each individual example to its own directory. 
* Move the code in the readme to bash scripts. 

The only two new files are `run_pl_ner.py` and `transformers_base.py`.

The goal is to keep the same format as the original command-line. Most of the argument names are preserved. I have verified that for NER the results of the same on GPU. 

There are several nice benefits of lightning -> somewhat nicer logging and library integration (e.g. wandb), auto-checkpointing. Mostly the goal though is code readability with identical functionality. 

Todo: 
* make sure that the output file format is identical. 
* print test results after training. 
* test multi-gpu and apex (in theory these should work) 

Comments:
Hi @srush, thanks for this PR :heart:  Can't wait to test it!

One suggestion/RFC: could we rename it to something like `token_classification` instead of `ner`. I know PoS tagging is not really covered in recent papers, but I always test new models for this task with the "identical" implementation 😅 This requires only a little modification in the code: we then should report accuracy as well.

But I will be totally fine with `ner` here!

Token Classification sounds good to me. That is consistent with the internal naming.
/pytorch/aten/src/THC/THCTensorScatterGather.cu:100: void THCudaTensor_gatherKernel(TensorInfo<Real, IndexType>, TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = 3]: block: [4,0,0], thread: [319,0,0] Assertion `indexValue >= 0 && indexValue < src.sizes[dim]` failed.
Author: DeepFool
Pull request: no
Labels: name: wontfix

## ❓ Questions & Help

<!-- A clear and concise description of the question. -->
I have no idea about this error。

Comments:
Could you please provide more information? Where does this error occur? Are you using one of our example scripts? I believe there are templates you can use so we may help you more efficiently.

> Could you please provide more information? Where does this error occur? Are you using one of our example scripts? I believe there are templates you can use so we may help you more efficiently.

I run run_squad.py in a chinese reading comprehension dataset. I change sereval places in utils_squad.py ,I throw all examples which have no answer because my passage is so longth. if i don't throw all examples without answer, my trained model will predict no answer in test dataset. This error occur in the begining of training. I have no idea why this error happen ?

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
ValueError: too many dimensions 'str'
Author: lenyabloko
Pull request: no

# 🐛 Bug

**To Reproduce**
Steps to reproduce the behavior:

Here is my Colab Notebook you can run to to see the error
https://colab.research.google.com/drive/1ESyf46RNBvrg-7DDQ5l8zhlKZjWGdqUv#scrollTo=MqlsdjFVMmMZ

```
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-9-0b9dcdf94c77> in <module>()
     71 
     72 # Train the model
---> 73 model.train_model(train_df)
     74 
     75 # Evaluate the model

1 frames
/usr/local/lib/python3.6/dist-packages/simpletransformers/classification/classification_model.py in train_model(self, train_df, multi_label, output_dir, show_running_loss, args, eval_df, verbose, **kwargs)
    261             ]
    262 
--> 263         train_dataset = self.load_and_cache_examples(train_examples, verbose=verbose)
    264 
    265         os.makedirs(output_dir, exist_ok=True)

/usr/local/lib/python3.6/dist-packages/simpletransformers/classification/classification_model.py in load_and_cache_examples(self, examples, evaluate, no_cache, multi_label, verbose, silent)
    757 
    758         if output_mode == "classification":
--> 759             all_label_ids = torch.tensor([f.label_id for f in features], dtype=torch.long)
    760         elif output_mode == "regression":
    761             all_label_ids = torch.tensor([f.label_id for f in features], dtype=torch.float)

ValueError: too many dimensions 'str'

```

The problem arises when using

```
from simpletransformers.classification import ClassificationModel

import pandas as pd
prefix = '/content/'
train_df = pd.read_csv(prefix + 'train.csv', header=None)
train_df=train_df.drop(index=0)

model = ClassificationModel('roberta', 'roberta-base')

model.train_model(train_df)

```



Comments:
I suggest to close this topic and keep the discussion over at https://github.com/ThilinaRajapakse/simpletransformers/issues/229.
BertModel' object missing 'save_pretrained' attribute
Author: ArashAskary
Pull request: no

I was attempting to download a pre-trained BERT model & save it to my cloud directory using Google Colab.

model.save_pretrained() seems to be missing completely for some reason. 

Link to Colab notebook: https://colab.research.google.com/drive/1ix_nNhsd89nLfTy6Nyh-Ak8PHn1SYm-0

Here's my code:

```
!pip install pytorch_pretrained_bert

import torch
from pytorch_pretrained_bert import BertTokenizer, BertModel 
import pandas as pd
import numpy as np


### Let's download a model and tokenizer
model = BertModel.from_pretrained('bert-base-uncased')
tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')

### Now let's save our model and tokenizer to a directory
model.save_pretrained('./models/')
tokenizer.save_pretrained('./models/')

```

Error:
```
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-18-1a3b2c8b8e82> in <module>()
     11 
     12 ### Now let's save our model and tokenizer to a directory
---> 13 model.save_pretrained('./models/')
     14 tokenizer.save_pretrained('./models/')
     15 

/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py in __getattr__(self, name)
    574                 return modules[name]
    575         raise AttributeError("'{}' object has no attribute '{}'".format(
--> 576             type(self).__name__, name))
    577 
    578     def __setattr__(self, name, value):
```

AttributeError: 'BertModel' object has no attribute 'save_pretrained'


Comments:
Hi, `from_pretrained` appeared in an older version of the library. `pytorch-pretrained-BERT` is a year old, is less robust and lacks certain functionalities (such as the one you mentioned) which are present in `transformers`.
Allow import of model components, e.g. `from transformers import TFAlbertMLMHead`
Author: jarednielsen
Pull request: no

# 🚀 Feature request

Expose model components such as custom layers like `TFAlbertMLMHead`, that are created in `modeling_tf_albert.py` and others.

## Motivation

I'm writing a custom pretraining script that incorporates both the masked language modeling (MLM) and sentence order prediction (SOP) objectives. I'm able to use the `TFAlbertForMaskedLM` model for the MLM objective, but need access to the `last_hidden_state` to write my SOP objective. I can do this if I have a raw `TFAlbertModel` and write my own MLM objective, but would prefer to just create my own model from pre-existing modularized components.

## Your contribution

I can contribute this, it would just be modifying `__init__.py`. I know that there's a lot of care taken in API design, and the team may have explicitly decided against this. But if it is an option, it would make the transformers repo much more extensible for research.



Comments:
Why don't you import it directly via
`from transformers.modeling_tf_albert import TFAlbertMLMHead`
?
In my opinion it is not good practise to expose everything via `__init__.py` because the  autocomplete feature of an IDE will become messy.

That's true, don't know why that slipped my mind. Thanks for the suggestion!
how to train RoBERTa from scratch
Author: 008karan
Pull request: no
Labels: name: wontfix

## ❓ Questions & Help

<!-- A clear and concise description of the question. -->
I want to train RoBERTa model from scratch on different language. Is there any implementation available here to do this?

Comments:
https://github.com/pytorch/fairseq/blob/master/examples/roberta/README.pretraining.md

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


You can now leave `--model_name_or_path` to None in `run_language_modeling.py` to train a model from scratch.

See also https://huggingface.co/blog/how-to-train

When I put new --config_name and --tokenizer_name. It shows me that 
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
Anyone can help me?
How to do the further pretraining ?
Author: JiangYanting
Pull request: no
Labels: name: wontfix

## ❓ Questions & Help

<!-- A clear and concise description of the question. -->
Hello ! How could I futher Pre-train the BERT ( including the unsupervised masked language model
and next sentence prediction tasks ) **using my own corpus** ?   thank you very much !

Comments:
We have no scripts for pre-training, but we do have scripts for fine-tuning (which seems to be what you want to do). Take a look at [run_lm_finetuning.py](https://github.com/huggingface/transformers/blob/master/examples/run_lm_finetuning.py) for more information. 

We don't have examples that do NSP however, as it was proven with RoBERTa to not be particularly useful for training. You'll have to code it yourself or find an implementation somewhere if you want to train on that loss.

@LysandreJik  I got it. thank you !

@JiangYanting If you want to do LM finetuning incl. NSP, you might wanna have a look at [FARM](https://github.com/deepset-ai/FARM). 
There's an example script [here](https://github.com/deepset-ai/FARM/blob/master/examples/lm_finetuning.py ).  

From our experience it depends a lot on the domain whether NSP makes sense. In some industry applications, we made good experience with also adding other auxiliary tasks in this phase of model training (e.g. an additional classification task for available tags of documents / sentences).



@tholor  Wow, that's so cool ! I would have a try after I take a final exam^_^. thank you very much !

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
is right?
Author: ARDUJS
Pull request: no
Labels: name: Ex: LM (Pretraining)

https://github.com/huggingface/transformers/blob/master/examples/run_language_modeling.py
in 225 row

![image](https://user-images.githubusercontent.com/20811685/74525332-d1480e00-4f5b-11ea-8918-ae22507de8e8.png)

# 10% of the time, we replace masked input tokens with random word
but write 0.5 
is ok?

Comments:
Hi @ARDUJS can you update your issue title to something more descriptive? Thanks!

Should be correct -> 80% masked, that means 20% is left. Using this 20% in 50 % the random word is used, 50% original token is kept. So both random word and original has an overall prob. of 10%.

Original BERT is using the same logic, see [here](https://github.com/google-research/bert/blob/master/create_pretraining_data.py#L391).
Evaluation result.txt path suggestion
Author: Compile7683
Pull request: no
Labels: name: wontfix

## 🚀 Feature

At pytorch-transformers/examples/**run_lm_finetuning**.py and **run_glue**.py
There is a line ```output_eval_file = os.path.join(eval_output_dir, "eval_results.txt")```

When setting evaluate_during_training **True**, `output_eval_file` will keep being overwritten.

I think `output_eval_file` can be assigned like

 `output_eval_file = os.path.join(eval_output_dir, prefix, "eval_results.txt")`

Meanwhile, in the main() function

```result = evaluate(args, model, tokenizer, prefix=global_step)```

change into 
```
result = evaluate(args, model, tokenizer, prefix=checkpoint.split('/')[-1] if checkpoint.find('checkpoint') != -1 else "")
```
Just a little suggestion 


Comments:
Yes, why not, do you want to submit a PR for that?

> Yes, why not, do you want to submit a PR for that?

Thanks~
By the way, is there any code formatting requirement or a contribution docs for developers?


This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


seems like the first checkpoint's eval results does not have a prefix?
when will add XLMRobertaForQuestionAnswering package
Author: ynebula
Pull request: no

I will study squad of multilingual.

I found that question answer package did include in run_squad.py.

I wanna release that package

Are you plan to release XLMRobertaForQuestionAnwering?

please let me know.

Comments:
It is pretty easy to add the code yourself since RobertaForQuestionAnswering is already implemented and XLMRobertaForQuestionAnswering is just a wrapper around it. 

Thank you for your answering.

I got your mention

I have question one more

Is it possible to learn XLM-Roberta data to Roberta
(XLM-Roberta data is https://github.com/pytorch/fairseq/tree/master/examples/xlmr)

If it is possible, can you show me how to set up.

please let me know.

I resolve the problem.

Thank you your answering