noamaanMulla-03/transformers-issues huggingface dataset:
[Question]: Why does model.__call__ return the loss too?
Author: dclong
Pull request: no
Labels: name: Discussion

name: wontfix

name: Core: Modeling

# ❓ Questions & Help

In PyTorch, model.__call__ returns the output tensor and users have to call a loss function to get the loss. I wonder why models in transformers doesn't follow this convention? Any specific reason?

Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Model card for bert-turkish-question-answering question-answering model
Author: lserinol
Pull request: yes
Labels: name: model card

First model card for bert-turkish-question-answering question-answering model

Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/4281?src=pr&el=h1) Report
> Merging [#4281](https://codecov.io/gh/huggingface/transformers/pull/4281?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/b290c32e1617fa74f44ccd8b83365fe764437be9&el=desc) will **decrease** coverage by `0.00%`.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/4281/graphs/tree.svg?width=650&height=150&src=pr&token=9qOlN6Hb1c)](https://codecov.io/gh/huggingface/transformers/pull/4281?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master    #4281      +/-   ##
==========================================
- Coverage   78.39%   78.39%   -0.01%     
==========================================
  Files         120      120              
  Lines       19932    19932              
==========================================
- Hits        15626    15625       -1     
- Misses       4306     4307       +1     
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/4281?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [src/transformers/modeling\_tf\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/4281/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ190Zl91dGlscy5weQ==) | `92.77% <0.00%> (-0.17%)` | :arrow_down: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/4281?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/4281?src=pr&el=footer). Last update [b290c32...2222fd5](https://codecov.io/gh/huggingface/transformers/pull/4281?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


Awesome, thanks for sharing! Did you fine-tune https://huggingface.co/dbmdz/bert-base-turkish-cased?

 cc @stefan-it 

[Model page](https://huggingface.co/lserinol/bert-turkish-question-answering)

> Awesome, thanks for sharing! Did you fine-tune https://huggingface.co/dbmdz/bert-base-turkish-cased?
> 
> cc @stefan-it

Yes
Why there is no TransfoXLForSequenceClassification class?
Author: acriptis
Pull request: no
Labels: name: Discussion

name: wontfix

name: Core: Modeling

Hello, huggingface! Thank you for the great work with doing neat interfaces for transformers family!

I'm analyzing performance of transformers on a task of texts classification (like SST-2 in GLUE). I found that many architectures has `<ModelName>ForSequenceClassification` class, But `transformerXL` has restricted set of auxiliary classes (https://github.com/huggingface/transformers/blob/master/src/transformers/modeling_transfo_xl.py). 

Is there any reasons for missing these useful class? Is it planned to be implemented or is there any restrictions for such implementation?





Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Running on GPU?
Author: dimitarsh1
Pull request: no

## ❓ Questions & Help

Hello, 

I have a straightforward question I think, which I am curious about. 
When I load a pretrained model and use it to tokenise and extract embeddings, is the model running on a GPU or CPU? The reason why I am asking is that using bert is very slow. In particular approximately 100 times slower than the https://pypi.org/project/bert-embedding/ 

Any ideas? 

Cheers,
Dimitar

<!-- A clear and concise description of the question. -->

Comments:
Hi!

When you're talking about extracting the embeddings using a pre-trained model, what are you talking about exactly?

Are you talking about using the tokenizer like :
```python
tokenizer.encode(text)
```
 which returns the word ids?

Are you talking about using the embedding layer inside the model like :
```python
model.embeddings.word_embeddings(value)
```

Or are you talking about the encoded representation returned by the transformer after a forward pass like
```
model(value)
``` 

For the first one, using the tokenizer, you are simply using a python dictionary so it will run on CPU. For the next two, it depends on where you put your model. If you simply loaded it, it will be on CPU, but if you put in on a specific device using `model.to(device)`, then it will be on the specified device.

Yes, thank you. I messed up the model. 

 summarizer_cnn = pipeline('summarization')
summary_cnn = summarizer_cnn(sum_data)

#where sum_data is a textual data of 1000 length.

When I load a pretrained model and use it to extract summary, the model is running on a CPU instead of CPU and that is the reason bert is very slow. how to run the pretrained model and script on GPU.
TAPAS: Weakly Supervised Table Parsing via Pre-training
Author: TheEdoardo93
Pull request: no
Labels: name: New model

# 🌟 TAPAS (Weakly Supervised Table Parsing via Pre-Training)

## Model description

**TAPAS** [ArXiV Paper](https://arxiv.org/pdf/2004.02349.pdf) [Google AI Blog](https://ai.googleblog.com/2020/04/using-neural-networks-to-find-answers.html?utm_campaign=Learning%20Posts&utm_content=128690771&utm_medium=social&utm_source=linkedin&hss_channel=lcp-9358547), an approach to question answering over tables without generating logical forms. TAPAS trains from weak supervision, and predicts the denotation by selecting table cells and optionally applying a corresponding aggregation operator to such selection. TAPAS extends BERT’s architecture to encode tables as input, initializes from an effective joint pre-training of text segments and tables crawled from Wikipedia, and is trained end-to-end. We experiment with three different semantic parsing datasets, and find that TAPAS outperforms or rivals semantic parsing models by improving state-of-the-art accuracy on SQA from 55.1 to 67.2 and performing on par with the state-of-the-art on WIKISQL and WIKITQ, but with a simpler model architecture. We additionally find that transfer learning, which is trivial in our setting, from WIKISQL to WIKITQ, yields 48.7 accuracy, 4.2 points above the state-of-the-art.

## Open source status

* [ X ] the model implementation is available: [GitHub repo](https://github.com/google-research/tapas/tree/master/tapas/models)
* [ X ] the model weights are available: in the "Data" section at [their GitHub page](https://github.com/google-research/tapas)
* [ X ] who are the authors: Jonathan Herzig, Paweł Krzysztof Nowak, Thomas Muller, Francesco Piccinno, Julian Martin Eisenschlos

Comments:
Duplicated of [https://github.com/huggingface/transformers/issues/4166](url)
[model_cards]: 🇹🇷 Add new ELECTRA small and base models for Turkish
Author: stefan-it
Pull request: yes
Labels: name: model card

Hi,

this PR introduces model cards for new ELECTRA small and base models for Turkish 🇹🇷.

More information (checkpoint evaluation, downstream evaluation on Pos Tagging and NER, loss curves, TensorBoards) see [this repository](https://github.com/stefan-it/turkish-bert/tree/electra/electra).

Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/4318?src=pr&el=h1) Report
> Merging [#4318](https://codecov.io/gh/huggingface/transformers/pull/4318?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/4bf5042240d33286460b83f3dbf9be77500faab3&el=desc) will **increase** coverage by `0.01%`.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/4318/graphs/tree.svg?width=650&height=150&src=pr&token=9qOlN6Hb1c)](https://codecov.io/gh/huggingface/transformers/pull/4318?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master    #4318      +/-   ##
==========================================
+ Coverage   78.16%   78.17%   +0.01%     
==========================================
  Files         120      120              
  Lines       20005    20005              
==========================================
+ Hits        15636    15638       +2     
+ Misses       4369     4367       -2     
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/4318?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [src/transformers/modeling\_tf\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/4318/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ190Zl91dGlscy5weQ==) | `92.93% <0.00%> (+0.16%)` | :arrow_up: |
| [src/transformers/trainer.py](https://codecov.io/gh/huggingface/transformers/pull/4318/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy90cmFpbmVyLnB5) | `41.09% <0.00%> (+0.28%)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/4318?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/4318?src=pr&el=footer). Last update [4bf5042...dc686bb](https://codecov.io/gh/huggingface/transformers/pull/4318?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
Update README.md
Author: savasy
Pull request: yes
Labels: name: model card


Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/4315?src=pr&el=h1) Report
> Merging [#4315](https://codecov.io/gh/huggingface/transformers/pull/4315?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/4bf5042240d33286460b83f3dbf9be77500faab3&el=desc) will **increase** coverage by `0.00%`.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/4315/graphs/tree.svg?width=650&height=150&src=pr&token=9qOlN6Hb1c)](https://codecov.io/gh/huggingface/transformers/pull/4315?src=pr&el=tree)

```diff
@@           Coverage Diff           @@
##           master    #4315   +/-   ##
=======================================
  Coverage   78.16%   78.16%           
=======================================
  Files         120      120           
  Lines       20005    20005           
=======================================
+ Hits        15636    15637    +1     
+ Misses       4369     4368    -1     
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/4315?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [src/transformers/modeling\_tf\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/4315/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ190Zl91dGlscy5weQ==) | `92.77% <0.00%> (ø)` | |
| [src/transformers/trainer.py](https://codecov.io/gh/huggingface/transformers/pull/4315/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy90cmFpbmVyLnB5) | `41.09% <0.00%> (+0.28%)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/4315?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/4315?src=pr&el=footer). Last update [4bf5042...ef8a2ed](https://codecov.io/gh/huggingface/transformers/pull/4315?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
Improvements to the wandb integration
Author: vanpelt
Pull request: yes

Hey guys,

I'm one of the founders of W&B and have been working with @borisdayma on the wandb integration. I made a few changes to address a few cases we thought through:

1. If wandb is installed but the user hasn't logged in we now print a warning instead of prompting for an api_key
2. Made messaging more clear on how to install wandb and linked to more documentation
3. Added an environment variable for disabling wandb. This is important for users using fastai or lightning with their own wandb callback.
4. Added an environment variable for disabling wandb.watch. Because we're pulling gradients and parameters during training there is a small performance impact on training. We wanted to make it easy for users to disable this feature.

Happy to iterate on any feedback you guys have. Really excited about this integration!

Comments:
Hey guys, any updates on this?

@vanpelt Can't push to your fork so will tweak things in #4324 and combine with @borisdayma's #4221
Feature Request: Pipeline for Query/Document relevance
Author: ArthurCamara
Pull request: no
Labels: name: wontfix

name: Core: Pipeline

# Pipelines for IR tasks

## Justification

In the last few years, a bunch of deep architectures were proposed for Ad-hoc retrieval, most with limited success (if any). However, BERT(et al)-based models are finally pushing the state of the art for Ad-hoc retrieval. In fact in the last TREC had a [Deep Learning track](https://microsoft.github.io/TREC-2019-Deep-Learning/) whre "NNLM" (neural network language models) [dominated](https://twitter.com/UnderdogGeek/status/1206595356017848324/photo/1) any other traditional (Mostly BM25 and variations) and other deep models. 

So, it's a current trend that BERT should be the new baseline for any proposed model for IR.

## Description

There should be a [pipeline-like](https://github.com/huggingface/transformers#quick-tour-of-pipelines) feature that is able to score pairs of documents and user queries. Probably, pre-trained on a dataset like the [MSMarco dataset for TREC'19](https://microsoft.github.io/TREC-2019-Deep-Learning/). Ideally, this would also support a list of documents to rank and return scores.

In real-life applications, one would probably want to combine BERT scores with a traditional baseline scores (like QL or BM25). So, the score is needed (or, even better, combine something like [pyserini](https://github.com/castorini/pyserini) in the backend?)

## Usage
```
from transformers import pipeline

# Allocate a pipeline for sentiment-analysis
nlp = pipeline('document-relevancy')
nlp({
    'query 'can hives be a sign of pregnancy',
    'context' '<document content>'
})
>>> {'score': 0.28756016668193496'}
```

I have already used DistilBERT on a paper to appear on ECIR2020 (Diagnosing BERT with Retrieval Heuristics), and would be able to contribute the model for this (even for bert-base).

I would also love to contribute with this, but will probably need some guidance, if anyone is willing to help.

Thanks!

Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


unstale because this is very interesting

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
urgent - ROBERTA on WSC
Author: yes1234man
Pull request: no
Labels: name: wontfix

Hi
I am looking for how to run ROBERTA on WSC data, similar to the example in fairseq:
https://github.com/pytorch/fairseq/tree/master/examples/roberta

However, fairseq is hard to use and modify and I really appreciate if you could add this to your great repo
thanks 

Comments:
The conversion script should pretty much work out of the box, so feel free to do it and we welcome a PR (and we can upload the converted weights to our S3)

which script you mean? sorry i did not get it. i think this is not possible
to train it on huggingface is this?

On Fri, Mar 6, 2020, 10:12 PM Julien Chaumond <notifications@github.com>
wrote:

> The conversion script should pretty much work out of the box, so feel free
> to do it and we welcome a PR (and we can upload the converted weights to
> our S3)
>
> —
> You are receiving this because you authored the thread.
> Reply to this email directly, view it on GitHub
> <https://github.com/huggingface/transformers/issues/3161?email_source=notifications&email_token=AODM7I44VQEQWKUWI5SYT3TRGFRMLA5CNFSM4LDD7F42YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEOC335Y#issuecomment-595967479>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AODM7I6VCDKQWMAKL4XDHETRGFRMLANCNFSM4LDD7F4Q>
> .
>


This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
How can I run NER on ALBERT?
Author: xf05888
Pull request: no
Labels: name: wontfix

name: Ex: Named Entity Recognition

I what to run NER on ALBERT, so I checked the [run_ner.py](https://github.com/huggingface/transformers/blob/master/examples/run_ner.py), but it seems like no ALBERT support. 
So can I simply import `AlbertTokenizer`, `AlbertForTokenClassification` and
`AlbertConfig` in the script and add them to `MODEL_CLASSES` and `ALL_MODELS`(Or need other config)?

Comments:
+1 
Same for [run_language_modeling.py](https://github.com/huggingface/transformers/blob/master/examples/run_language_modeling.py)?

What do you think about moving the examples to `AutoModels` (in this case `AutoModelForTokenClassification`) @srush @LysandreJik @julien-c ?

@thomwolf Indeed, that would be nice.

Yup, sounds good to me (it will make things much simpler). 

Yes, makes sense

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
fast gpt2 inference
Author: rajarsheem
Pull request: no
Labels: name: wontfix

I have a fine-tuned ```GPT2LMHeadModel``` (gpt2-medium) which I am using to run inference on large data (>60M sequences) offline. At each iteration, my input is a batch of 30 variable-length sequences which gets padded according to the max length of the batch. My current speed is around 8 secs/iter and input sequences have around 150 tokens.

I am using pytorch's ```BucketIterator``` to group sequences of similar length and minimize padding. I am using fp-16. These are increasing the throughput. The GPU I am using is Tesla V100.

Can you please suggest what all other optimizations that I can do to increase the speed?

Comments:
Have given some thought to using TorchScript but since my input sequence length changes each time, the easier tracing approach won't work.

ONNX also faced the same problem.

Hi rajarsheem,
Can you please share your code that doing the batch inference with variable-length sequences?

Thanks

Is gpt2 model traceable? 

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
[bug in run_glue.py] GlueDataset with no local_rank when init
Author: jia-zhuang
Pull request: no

https://github.com/huggingface/transformers/blob/241759101e7104192d01a07fc70432fa02ae8cb7/examples/text-classification/run_glue.py#L137

GlueData need to add local_rank parameter when init

```python
train_dataset = GlueDataset(data_args, tokenizer=tokenizer, local_rank=training_args.local_rank) if training_args.do_train else None
```

Comments:
Not on master.

You should install from source when running the examples, as specified in the [README](https://github.com/huggingface/transformers#run-the-examples).
Fix for #3846
Author: lukovnikov
Pull request: yes

Fix for #3846 . PretrainedTokenizer mapped " do not" to " don't" when .decode(...) is called. Removed the " do not" --> " don't" mapping from clean_up_tokenization(...).

Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/4024?src=pr&el=h1) Report
> Merging [#4024](https://codecov.io/gh/huggingface/transformers/pull/4024?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/ab90353f1abfd15f8d21f99395658d060679a08c&el=desc) will **decrease** coverage by `0.08%`.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/4024/graphs/tree.svg?width=650&height=150&src=pr&token=9qOlN6Hb1c)](https://codecov.io/gh/huggingface/transformers/pull/4024?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master    #4024      +/-   ##
==========================================
- Coverage   78.01%   77.93%   -0.09%     
==========================================
  Files         114      114              
  Lines       18671    18671              
==========================================
- Hits        14566    14551      -15     
- Misses       4105     4120      +15     
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/4024?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [src/transformers/tokenization\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/4024/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy90b2tlbml6YXRpb25fdXRpbHMucHk=) | `89.36% <ø> (ø)` | |
| [src/transformers/modeling\_tf\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/4024/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ190Zl91dGlscy5weQ==) | `90.47% <0.00%> (-2.47%)` | :arrow_down: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/4024?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/4024?src=pr&el=footer). Last update [ab90353...333778c](https://codecov.io/gh/huggingface/transformers/pull/4024?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


This looks good to me :-) Thanks for the PR!  What do you think @thomwolf @n1t0 ?

Hi @lukovnikov,

was the last commit by accident? I don't think that this one is needed to fix the issue no?

Thanks for pointing that out, should be good now.