noamaanMulla-03/transformers-issues huggingface dataset:
RoBERTaMultiChoice does not work with `roberta-large`
Author: yuchenlin
Pull request: no

# 🐛 Bug

## Information

Model I am using (Bert, XLNet ...):
**roberta-large**
Language I am using the model on (English, Chinese ...):
English 
The problem arises when using:
* [* ] the official example scripts: 
https://github.com/huggingface/transformers/tree/6c1b23554f8bb5b5e1f6c80969acab764c755678/examples#multiple-choice
* [ ] my own modified scripts: (give details below)

The tasks I am working on is:
* [ *] an official GLUE/SQUaD task: **SWAG**
* [ ] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:
just follow the example code of running swag dataset but with `robert-large` instead of `roberta-base` (which works well)

```
export SWAG_DIR=~/swagaf-master/data/
python ./examples/run_multiple_choice.py \
--model_type roberta \
--task_name swag \
--model_name_or_path roberta-large \
--do_train \
--do_eval \
--do_lower_case \
--data_dir $SWAG_DIR \
--learning_rate 5e-5 \
--num_train_epochs 3 \
--max_seq_length 80 \
--output_dir tmp/swag_base \
--per_gpu_eval_batch_size=16 \
--per_gpu_train_batch_size=16 \
--gradient_accumulation_steps 2 \
--overwrite_output
```

And it will say:

02/08/2020 00:46:23 - INFO - transformers.modeling_utils -   **Weights of RobertaForMultipleChoice not initialized from pretrained model:** ['classifier.weight', 'classifier.bias']

02/08/2020 00:46:23 - INFO - transformers.modeling_utils -   **Weights from pretrained model not used in RobertaForMultipleChoice:** ['lm_head.bias', 'lm_head.dense.weight', 'lm_head.dense.bias', 'lm_head.layer_norm.weight', 'lm_head.layer_norm.bias', 'lm_head.decoder.weight']


Consequently, the script is to learn a model from the scratch instead of fine-tuning pre-trained roberta-large.

## Expected behavior

<!-- A clear and concise description of what you would expect to happen. -->
It should load the pre-trained weights of roberta-large model.

## Environment info
<!-- You can run the command `python transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->
     
- `transformers` version: 2.4.1
- Platform: linux
- Python version: 3.7
- PyTorch version (GPU?): GPU
- Tensorflow version (GPU?): n/a
- Using GPU in script?: no
- Using distributed or parallel set-up in script?: no


Comments:
@yuchenlin May I ask if you figured out the bug?
SequenceSummary: config.summary_activation = 'relu' would be ignored
Author: sshleifer
Pull request: no
Labels: name: Ex: Sequence Classification

name: Core: Modeling

This isn't a bug, but merely an unintuitive argument name.

`summary_activation` sounds like it can be general, e.g. "relu" or "gelu" or something, but if its not "tanh" its ignored.

Since I assume it's annoying to go through all the configs and rename a field to use_tanh=True, I propose that we raise if summary_activation is a string thats not tanh, instead of silently just using no activation.

Another approach could integrate an `ACT2FN` dictionary (see  https://github.com/huggingface/transformers/issues/1347)
to actually support the other activation functions.

Happy to do either approach if others think it would be useful.


Comments:
I think both approaches are reasonable. @LysandreJik @thomwolf?

I agree with both approaches as well. The second one would probably be the most useful.

Yes, like Lysandre
Add more specific testing advice to Contributing.md
Author: sshleifer
Pull request: yes


Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/2815?src=pr&el=h1) Report
> Merging [#2815](https://codecov.io/gh/huggingface/transformers/pull/2815?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/bed38d3afec99ce99ef8610337cb279a8fb25033?src=pr&el=desc) will **increase** coverage by `0.55%`.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/2815/graphs/tree.svg?width=650&token=9qOlN6Hb1c&height=150&src=pr)](https://codecov.io/gh/huggingface/transformers/pull/2815?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master    #2815      +/-   ##
==========================================
+ Coverage   75.02%   75.58%   +0.55%     
==========================================
  Files          93       93              
  Lines       15275    15275              
==========================================
+ Hits        11460    11545      +85     
+ Misses       3815     3730      -85
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/2815?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [src/transformers/data/processors/squad.py](https://codecov.io/gh/huggingface/transformers/pull/2815/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9kYXRhL3Byb2Nlc3NvcnMvc3F1YWQucHk=) | `56.49% <0%> (+27.59%)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/2815?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/2815?src=pr&el=footer). Last update [bed38d3...4ae60cb](https://codecov.io/gh/huggingface/transformers/pull/2815?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
import Error from official example caused by fastprogress
Author: karajan1001
Pull request: no

## 🐛 Bug

<!-- Important information -->

Model I am using (Bert, XLNet....): BERT

Language I am using the model on (English, Chinese....): ALL

The problem arise when using:
* [x] the official example scripts: (give details)
* [ ] my own modified scripts: (give details)

The tasks I am working on is:
* [x] an official GLUE/SQUaD task: (give the name)
* [ ] my own task or dataset: (give details)

## To Reproduce

Steps to reproduce the behavior:

V0.2.1 of fastprogress released a couple of days ago seems to cause errors in run_tf_ner.py in the official example.

Traceback (most recent call last):
  File "run_tf_ner.py", line 12, in <module>
    from fastprogress import master_bar, progress_bar
ImportError: cannot import name 'master_bar' from 'fastprogress' (/usr/local/lib/python3.7/dist-packages/fastprogress/__init__.py)

users need to either downgrade: pip3 install fastprogress==0.1.22
or change the code: 
` 
from fastprogress.fastprogress import master_bar, progress_bar
`


## Expected behavior

<!-- A clear and concise description of what you expected to happen. -->

## Environment

* OS:
* Python version:
* PyTorch version:
* PyTorch Transformers version (or branch):
* Using GPU ?
* Distributed of parallel setup ?
* Any other relevant information:

## Additional context

<!-- Add any other context about the problem here. -->


Comments:
Thanks for reporting. (and hello @sgugger :)

I'll merge this to fix the immediate issue, but maybe @jplu can chime in: maybe we don't need the fastprogress dependency here?

Closed by #2400 

Oh, I forgot to update the `__init__` with the new version. Will add back the functions there to make compatibility easier. Thanks for the super quick fix!

Hello!! Thanks @julien-c for pinging me :)

Indeed my code was not compatible with the last version of fastprogress, but I thought to have specified the version in the `requirements.txt` file but apparently the way of installing the transformers framework has changed recently.

@sgugger good job, I like your fix :)

@julien-c fastprogress if (in my opinion) the most convenient progress bar to use for model training, but I can change if it becomes a problem, as you wish.


Alright let's use `fastprogress` then! We can clean up the conditional import down the line.

I change the:
`from fastprogress import master_bar, progress_bar` 
to 
`from fastprogress.fastprogress import master_bar, progress_bar`
in the  ~/fastai/imports/core.py file and it worked
'distilbert-base-cased-distilled-squad' was not found error
Author: elronbandel
Pull request: no

# 🐛 Bug

## Information

Model I am using: distilbert-base-cased-distilled-squad

The problem arises when using: AutoTokenizer or AutoModelForQuestionAnswering

Steps to reproduce the behavior:
0. make sure you have everything on colab installed and imported 
```
!pip install transformers
import torch
from transformers import AutoModelForQuestionAnswering, AutoTokenizer
```
1. run the code on google colab:
```

tokenizer = AutoTokenizer.from_pretrained("distilbert-base-cased-distilled-squad")

model = AutoModelForQuestionAnswering.from_pretrained("distilbert-base-cased-distilled-squad")
```

and the error:
```
OSError                                   Traceback (most recent call last)
/usr/local/lib/python3.6/dist-packages/transformers/configuration_utils.py in get_config_dict(cls, pretrained_model_name_or_path, pretrained_config_archive_map, **kwargs)
    224             if resolved_config_file is None:
--> 225                 raise EnvironmentError
    226             config_dict = cls._dict_from_json_file(resolved_config_file)

OSError: 

During handling of the above exception, another exception occurred:

OSError                                   Traceback (most recent call last)
3 frames
/usr/local/lib/python3.6/dist-packages/transformers/configuration_utils.py in get_config_dict(cls, pretrained_model_name_or_path, pretrained_config_archive_map, **kwargs)
    239                     )
    240                 )
--> 241             raise EnvironmentError(msg)
    242 
    243         except json.JSONDecodeError:

OSError: Model name 'distilbert-base-cased-distilled-squad' was not found in model name list. We assumed 'https://s3.amazonaws.com/models.huggingface.co/bert/distilbert-base-cased-distilled-squad/config.json' was a path, a model identifier, or url to a configuration file named config.json or a directory containing such a file but couldn't find any such file at this path or url.
```


## Environment info     
- `transformers` version: 2.4.1
- Platform: Google Colab
- PyTorch version :1.4.0
- Using GPU in script?: no
- Using distributed or parallel set-up in script?: no


Comments:
Hi! This checkpoint was added six days ago but our latest release was 13 days ago, so you would need to install the repository from source to use that model:

```
pip install git+https://github.com/huggingface/transformers
```

It'll be available in a pip install once we do a new release.
How can I get similarity matching ?
Author: dimwael
Pull request: no
Labels: name: wontfix

## ❓ Questions & Help
Is there any way that can help me calculate the similarity between 2 questions ? Sometimes the questions is out of the scope of the data set questions. Using a simple similarity algorithm will always return the most similar even if it is not really correct. 
It is the same thing as here : 
https://github.com/deepmipt/dp_notebooks/blob/master/DP_BERT.ipynb

from deeppavlov import build_model, configs
model = build_model(configs.squad.squad_bert, download=True)
model(['DeepPavlov is a library for NLP and dialogue systems.'], ['What is DeepPavlov?'])

<!-- A clear and concise description of the question. -->


Comments:
Not sure to understand what you mean by `Using a simple similarity algorithm will always return the most similar even if it is not really correct`. What kind of simple similarity algo are you evoking here? What do you mean those simple algorithms aren't precise enough for your usecase?

Considering sentence similarity algorithms, I know:
- Statistical approach using bag-of-words TF-IDF-based methods like BM25 (better on longer docs than sentences).
- bag-of-word sentences and pooling (average-like) on word embeddings (word2vec-like) weighted by TF-IDF on a corpus for example.
- Full Sentence Embedding learnt directly on a similarity training set (maybe finetuned on your domain). It builds sentence embedding in a vector-space in which you can compute distance between sentences. Those models often uses siamese approach based on pre-trained language models such as BERT.

Sentence embedding technique requires more work and domain knowledge but is the one reaching the highest metrics in SOTA. Other techniques can be enough depending on your needs and domain.



Excuse me for not being clear enough, 
I wanted to say that if I have an FAQ data set and I want to get the most similar question for the user's question then, cosine similarity or TF-IDF will always give back a question even if it is not related. 
This is about classifying either that question does have a similar one or not like does the snippet of code I posted earlier from deeppavlov. 

No need to excuse ;)
Except an approach based on a dataset classifying "similar and non-similar" on your domain, any approach based on a score will require that you set a threshold to discriminate similar and non-similar. It can be fuzzy but sentence similarity is a very relative concept in any case.




This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Update with additional information
Author: iliaschalkidis
Pull request: yes

Added a "Pre-training details" section

Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/2852?src=pr&el=h1) Report
> Merging [#2852](https://codecov.io/gh/huggingface/transformers/pull/2852?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/1eec69a90007b8f4a7af10805dab4904ea5dea77?src=pr&el=desc) will **decrease** coverage by `1.07%`.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/2852/graphs/tree.svg?width=650&token=9qOlN6Hb1c&height=150&src=pr)](https://codecov.io/gh/huggingface/transformers/pull/2852?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master    #2852      +/-   ##
==========================================
- Coverage   75.06%   73.98%   -1.08%     
==========================================
  Files          94       94              
  Lines       15287    15287              
==========================================
- Hits        11475    11310     -165     
- Misses       3812     3977     +165
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/2852?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [src/transformers/modeling\_tf\_pytorch\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/2852/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ190Zl9weXRvcmNoX3V0aWxzLnB5) | `8.72% <0%> (-81.21%)` | :arrow_down: |
| [src/transformers/modeling\_roberta.py](https://codecov.io/gh/huggingface/transformers/pull/2852/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ19yb2JlcnRhLnB5) | `85.91% <0%> (-9.86%)` | :arrow_down: |
| [src/transformers/modeling\_xlnet.py](https://codecov.io/gh/huggingface/transformers/pull/2852/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ194bG5ldC5weQ==) | `70.86% <0%> (-2.3%)` | :arrow_down: |
| [src/transformers/modeling\_ctrl.py](https://codecov.io/gh/huggingface/transformers/pull/2852/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ19jdHJsLnB5) | `92.07% <0%> (-2.21%)` | :arrow_down: |
| [src/transformers/modeling\_openai.py](https://codecov.io/gh/huggingface/transformers/pull/2852/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ19vcGVuYWkucHk=) | `80.2% <0%> (-1.35%)` | :arrow_down: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/2852?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/2852?src=pr&el=footer). Last update [1eec69a...59baea0](https://codecov.io/gh/huggingface/transformers/pull/2852?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


See #2851.

Thanks!
CUDA out of memory issue in the middle of training in run_language_modeling.py (say after 1000 steps).
Author: yuvalkirstain
Pull request: no
Labels: name: Need more information

name: Ex: LM (Finetuning)

# 🐛 Bug
CUDA OOM in run_language_modeling.py after many steps.
## Information
It seems strange to get them so late in the training procedure.
Model I am using (Bert, XLNet ...):
roberta-large
Language I am using the model on (English, Chinese ...):
English
The problem arises when using:
* [ ] the official example scripts: run_language_modeling.py, with line_by_line parameter

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: Wikitext after some filter of sentences (attached)
[wiki.train.raw.time_filter.normalized.text.txt](https://github.com/huggingface/transformers/files/4191154/wiki.train.raw.time_filter.normalized.text.txt)

## To reproduce
just run the script with the line_by_line parameter
Steps to reproduce the behavior:

- `transformers` version: latest
- Platform:
- Python version: 3.7.0
- PyTorch version (GPU?): latest
- Tensorflow version (GPU?): 
- Using GPU in script?: yes - TITAN Xp
- Using distributed or parallel set-up in script?: no


Comments:
what is your GPU?

> what is your GPU?

TITAN Xp


If I'm not mistaken the Titan XP has 12GB of VRAM? From my tests training RoBERTa-large with a batch size of 1 already requires 10GB of VRAM, so your GPU memory should be filled quickly. It is surprising that it crashes later though. Could you try with a smaller batch size and gradient accumulation?

Working with a smaller batch size and gradient accumulation works better. Thank you !
solve the exception: [AttributeError: 'bool' object has no attribute 'mean']
Author: 7will10
Pull request: yes

modified method simple_accuracy(),
before:
it's (preds == labels).mean()
This will cause an exception[AttributeError: 'bool' object has no attribute 'mean'],
then after update:
change to accuracy_score(labels,preds),
use this method accuracy_score() in package sklearn.metrics

Comments:
Hi! I'd like to replicate the error you had with the `AttributeError`. Could you let me know in which situation you faced this error?
Fix typos
Author: iliaschalkidis
Pull request: yes


Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/2857?src=pr&el=h1) Report
> Merging [#2857](https://codecov.io/gh/huggingface/transformers/pull/2857?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/925a13ced1e155ea7e55e14e177a7b5ae7ad174c?src=pr&el=desc) will **not change** coverage.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/2857/graphs/tree.svg?width=650&token=9qOlN6Hb1c&height=150&src=pr)](https://codecov.io/gh/huggingface/transformers/pull/2857?src=pr&el=tree)

```diff
@@           Coverage Diff           @@
##           master    #2857   +/-   ##
=======================================
  Coverage   75.06%   75.06%           
=======================================
  Files          94       94           
  Lines       15287    15287           
=======================================
  Hits        11475    11475           
  Misses       3812     3812
```



------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/2857?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/2857?src=pr&el=footer). Last update [925a13c...acca7c4](https://codecov.io/gh/huggingface/transformers/pull/2857?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
Fix typo
Author: mrm8488
Pull request: yes


Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/2855?src=pr&el=h1) Report
> Merging [#2855](https://codecov.io/gh/huggingface/transformers/pull/2855?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/925a13ced1e155ea7e55e14e177a7b5ae7ad174c?src=pr&el=desc) will **not change** coverage.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/2855/graphs/tree.svg?width=650&token=9qOlN6Hb1c&height=150&src=pr)](https://codecov.io/gh/huggingface/transformers/pull/2855?src=pr&el=tree)

```diff
@@           Coverage Diff           @@
##           master    #2855   +/-   ##
=======================================
  Coverage   75.06%   75.06%           
=======================================
  Files          94       94           
  Lines       15287    15287           
=======================================
  Hits        11475    11475           
  Misses       3812     3812
```



------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/2855?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/2855?src=pr&el=footer). Last update [925a13c...c86fc74](https://codecov.io/gh/huggingface/transformers/pull/2855?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
Added model card for bert-base-multilingual-uncased-sentiment
Author: yvespeirsman
Pull request: yes
Labels: name: model card

Added the model card for nlptown/bert-base-multilingual-uncased-sentiment

Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/2859?src=pr&el=h1) Report
> Merging [#2859](https://codecov.io/gh/huggingface/transformers/pull/2859?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/925a13ced1e155ea7e55e14e177a7b5ae7ad174c?src=pr&el=desc) will **not change** coverage.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/2859/graphs/tree.svg?width=650&token=9qOlN6Hb1c&height=150&src=pr)](https://codecov.io/gh/huggingface/transformers/pull/2859?src=pr&el=tree)

```diff
@@           Coverage Diff           @@
##           master    #2859   +/-   ##
=======================================
  Coverage   75.06%   75.06%           
=======================================
  Files          94       94           
  Lines       15287    15287           
=======================================
  Hits        11475    11475           
  Misses       3812     3812
```



------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/2859?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/2859?src=pr&el=footer). Last update [925a13c...917aa8d](https://codecov.io/gh/huggingface/transformers/pull/2859?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


@yvespeirsman Thanks for sharing! I can't push to your fork so I'll merge this and tweak it (languages have to be in a list)