noamaanMulla-03/transformers-issues huggingface dataset:
CircleCI ExamplesTests::test_run_squad failing
Author: sshleifer
Pull request: no

Started happening at  https://github.com/huggingface/transformers/pull/3428
and has been happening consistently.

scroll all the way down for [traceback](https://circleci.com/gh/huggingface/transformers/26044?utm_campaign=vcs-integration-link&utm_medium=referral&utm_source=github-build-link)

Comments:
![image](https://user-images.githubusercontent.com/6045025/77713019-b7163c80-6fab-11ea-9b78-0ed668943c18.png)

Happy to help on this @LysandreJik @patrickvonplaten

The test runs fine locally on my computer. A couple of wild thoughts:

- It happened after merging #3411 and #3428 which adds quite some long `t5-large` and `t5-base` tests to the test examples. Could this test fail because of some kind of time-out error?
- The test consumes 10GB RAM when running locally - this doesn't seem too much to fail though.

Did you try connecting to a failing circle-ci box, to investigate? 

unsubscribe




------------------&nbsp;原始邮件&nbsp;------------------
发件人:&nbsp;"Julien Chaumond"<notifications@github.com&gt;;
发送时间:&nbsp;2020年3月27日(星期五) 晚上10:15
收件人:&nbsp;"huggingface/transformers"<transformers@noreply.github.com&gt;;
抄送:&nbsp;"Subscribed"<subscribed@noreply.github.com&gt;;
主题:&nbsp;Re: [huggingface/transformers] CircleCI ExamplesTests::test_run_squad failing (#3469)



 Did you try connecting to a failing circle-ci box, to investigate?
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub, or unsubscribe.

I think this is solved with https://github.com/huggingface/transformers/pull/3485 .
T5 for summarization: pipeline x T5ForConditionalGeneration different results
Author: renatoviolin
Pull request: no

I've been doing experiments with text summarization and got some different results between pipeline and T5ForConditionalGeneration.

First I use the model.generate() to generate the resume. It runs very fast (even on CPU) and give poors results.

Second I use the pipeline, passing the same model I build on the first step. This runs slower and give a very good results.

Third, I re-run the first model.generate(). Now, the the model runs slower and produces the same result as pipeline.

I did a colab so that you can check.

I'm loosing any point in using the model x pipeline?

https://colab.research.google.com/drive/15HOerw3mYVCsjedW_dVGeyRX5cYWiNvS

Comments:
Hi @renatoviolin,

the T5 pipeline uses special input arguments for the `generate()` function that have been shown to work well for summarization. If you take a look at `task_specific_params` and under `summarization` in T5's config: https://s3.amazonaws.com/models.huggingface.co/bert/t5-base-config.json. You can see the `generate()` arguments that are used for pipeline.


Hi @patrickvonplaten 

Thanks for your attention. Now I got the point where I'm doing wrong.
But the most strange thing that happened is that in third step, after the pipeline was executed, the 
model.generate() produces similar results (and take long to run) as the pipeline.
At first glance, given the poor results and how quickly it runs, it seemed to me that the weights were not loaded.

I think it's probably because beam search is deactivated, no length penalties, no repeat penalties and a very short max length is used
Impossible to use T5 11b
Author: sdan
Pull request: no
Labels: name: Core: Modeling

# 🐛 Bug

## Information

Model I am using (Bert, XLNet ...):
T5 11B param
Language I am using the model on (English, Chinese ...):
English
The problem arises when using:
* [ x] the official example scripts: (give details below)
* [ ] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [x ] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:
```
summarizer = pipeline(task="summarization", model="t5-11b", tokenizer="t5-11b")
summary = summarizer(
	article,
	min_length=5, 
	max_length=100
	)
print("The Summary, ",summary[0]['summary_text'])
```

<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

## Expected behavior

As indicated in [T5's repo](https://github.com/google-research/text-to-text-transfer-transformer), they used Mesh Tensorflow, which is (according to them) the only way to run inference T5. This means the default CPU setting and even 1 GPU device setting would result in the following

1. CPU setting: run forever (my experience)
2. GPU setting: OOM (although I haven't tested this).

Meaning the current implementations of 3b and 11b would most likely render useless.

More testing needs to be done on whether this is the same for smaller models.

Get an output
<!-- A clear and concise description of what you would expect to happen. -->

## Environment info
<!-- You can run the command `python transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->
     
- `transformers` version: 2.7.0
- Platform: GCP
- Python version: 3.7.7
- PyTorch version (GPU?): 1.1.0 and no GPU
- Tensorflow version (GPU?):
- Using GPU in script?:
- Using distributed or parallel set-up in script?: no


Comments:
Update

On a regular CPU machine (not using GPUs), here's the benchmarks you'll need to load these models into memory and run them:

Bart-large-cnn (default): 5GB of RAM
T5-small: 14GB
T5-base: 20GB
T5-large: 31GB
T5-3b: 68GB
T5-11b: 120GB

So, I was initially wrong: **You can run this on CPU, but you'll need a lot of RAM, or try out GPUs**
Add shoarora/electra and alectra model cards
Author: shoarora
Pull request: yes
Labels: name: model card

Add model cards for recently uploaded models:

- shoarora/electra-small-owt (BERT)
- shoarora/alectra-small-owt (ALBERT)

Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/3524?src=pr&el=h1) Report
> Merging [#3524](https://codecov.io/gh/huggingface/transformers/pull/3524?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/33ef7002e17fe42b276dc6d36c07a3c39b1f09ed&el=desc) will **decrease** coverage by `0.01%`.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/3524/graphs/tree.svg?width=650&height=150&src=pr&token=9qOlN6Hb1c)](https://codecov.io/gh/huggingface/transformers/pull/3524?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master    #3524      +/-   ##
==========================================
- Coverage   77.80%   77.79%   -0.02%     
==========================================
  Files         100      100              
  Lines       17051    17051              
==========================================
- Hits        13267    13265       -2     
- Misses       3784     3786       +2     
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/3524?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [src/transformers/modeling\_tf\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/3524/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ190Zl91dGlscy5weQ==) | `88.15% <0.00%> (-0.18%)` | :arrow_down: |
| [src/transformers/modeling\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/3524/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ191dGlscy5weQ==) | `91.81% <0.00%> (-0.14%)` | :arrow_down: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/3524?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/3524?src=pr&el=footer). Last update [33ef700...daff82d](https://codecov.io/gh/huggingface/transformers/pull/3524?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


Model pages:
https://huggingface.co/shoarora/alectra-small-owt
https://huggingface.co/shoarora/electra-small-owt

Thanks for sharing @shoarora 

Did you see those models btw @LysandreJik?
Details on T5's current integration status 
Author: thomwolf
Pull request: no
Labels: name: wontfix

Hi all,

Regarding Google's T5 model, here is a quick summary of the status:
* the core model is in the library and some people have started to use it, but:
  - while the operations are identical or very similar (einsum vs. matmul), there is quite a significantly higher relative error between this model's PT hidden-state and the mesh-tensorflow hidden-state (in particular compared to our previous TF => PT model conversions).
  - our guess is that this comes for a combination of bfloat16 vs. fp32, einsum+model parallelism vs. matmul, plus the fact that we are not masking the hidden-states at each layer as the original implementation do (this should not matter much though).
  - as a consequence, we are waiting to be able to confirm it's performances on a GLUE fine-tuning before having a wider communication on its addition to the library.
* the full integration with GLUE tests requires a few features that we still need to add:
  - a decoding mechanism,
  - a pre/post-processing for GLUE to use it in text-to-text setting, and
  - a model parallelism feature
^^ we plan to work on these in February (from the more general view of having better encoder-decoder support in the library).

cc @julien-c @LysandreJik @sshleifer @patrickvonplaten

Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
XLNet model on S3 not set up correctly? 
Author: martinralfreindl
Pull request: no

Hi all, 

**I'm trying to initialize an XLNetForSequenceClassification model as follows:** 

```
from transformers import XLNetForSequenceClassification, XLNetConfig
configuration = XLNetConfig()
model = XLNetForSequenceClassification.from_pretrained("xlnet-base-cased", config=configuration)
```

**However I'm getting an error:** 

> RuntimeError: Error(s) in loading state_dict for XLNetForSequenceClassification:
> 	size mismatch for transformer.mask_emb: copying a param with shape torch.Size([1, 1, 768]) from checkpoint, the shape in current model is torch.Size([1, 1, 1024]).
> 	size mismatch for transformer.word_embedding.weight: copying a param with shape torch.Size([32000, 768]) from checkpoint, the shape in current model is torch.Size([32000, 1024]).
> 	size mismatch for transformer.layer.0.rel_attn.q: copying a param with shape torch.Size([768, 12, 64]) from checkpoint, the shape in current model is torch.Size([1024, 16, 64]).
>     ...

**I can get my code to run by specifying in my config:** 

```
configuration.d_model = 768 # hidden size --> should be 1024 
configuration.n_head = 12 # number of attention heads --> should be 16
configuration.d_inner = 3072 # FFN inner hidden size --> should be 4096
```

But I am somewhat confused: These adjustments to the model are not in line with XLNet as introduced in the [XLNet paper](https://arxiv.org/pdf/1906.08237.pdf) (page 13). Am I understanding something wrong here, or is the XLNet model on S3 not set up correctly? 

Comments:
After looking into this some more. It looks like my issue was arising from the fact that the XLNet authors published both a 'base' and 'large' version of their model: 

![image](https://user-images.githubusercontent.com/61122332/77469528-84cfd800-6e0f-11ea-8995-8a569bd2dc16.png)

It seems like the config defaults to the 'large' version, but I am loading the 'base' version. Adjustments to the parameters as outlined make sense in this light. 

There are simpler ways to do what you describe; first of all, you don't have to specify the configuration file, the model will load it automatically:

```py
from transformers import XLNetForSequenceClassification, XLNetConfig

model = XLNetForSequenceClassification.from_pretrained("xlnet-base-cased")
```

Secondly, you can also instantiate a configuration from a pre-trained checkpoint:

```py
from transformers import XLNetForSequenceClassification, XLNetConfig
configuration = XLNetConfig.from_pretrained("xlnet-base-cased")
model = XLNetForSequenceClassification.from_pretrained("xlnet-base-cased", config=configuration)
```
Document which heads are pretrained and which aren't
Author: BramVanroy
Pull request: no
Labels: name: wontfix

## 🚀 Feature

I was going through the documentation and I realised I never thought about the different heads in much detail (I always start from the base model and built on top of that). Now that I did, I wonder whether users (mistakenly?) assume that models such as `BertForQuestionAnswering` have a pretrained head. I am assuming that these heads are _not_ pretrained but that it is a convenience to have an architecture that can be finetuned on downstream tasks. If I am correct, it might be useful to highlight for these kind of models that the heads are not pretrained.

That being said, when I run 

```python
model = BertForSequenceClassification.from_pretrained('bert-base-uncased')
model.eval()
torch.set_grad_enabled(False)

for name, parameters in model.named_parameters():
    if 'classifier' in name:
        print(name)
        print(parameters)
```

I get the notice that "Weights of BertForSequenceClassification not initialized from pretrained model: ['classifier.weight', 'classifier.bias']", but still:

```
classifier.weight
Parameter containing:
tensor([[-0.0098,  0.0137,  0.0275,  ..., -0.0221, -0.0190,  0.0156],
        [ 0.0144,  0.0016,  0.0084,  ...,  0.0055,  0.0221, -0.0145]],
       requires_grad=True)
````

I've been peaking into the rabbit hole, but I can't seem to find where this random initialisation occurs.

If you agree, I can put some time in putting that in the documentation but I might need help or at least a review. Perhaps this can even be automated? That would be awesome.

Comments:
I'm certain the random initialisation occurs when we instantiate the class (See `BertPreTrainedModel.init_weights()`)

> I'm certain the random initialisation occurs when we instantiate the class (See `BertPreTrainedModel.init_weights()`)

You're right. It gets a bit complicated to track down though.

PretrainedModel implements `init_weights` which applies `self._init_weights` to all modules **but** there is no reference in that class to that method. You'll have to find it in the subclasses (e.g. BertPreTrainedModel). But for e.g. RoBERTa, it's of course not needed because RoBERTa extends BERT. It's a bit confusing to follow along - or at least it takes some time to get your head around.

It might be useful to implement the methods that are needed in subclasses in PreTrainedModel. Typically you'd see abstract methods for this, but something as simple as the following would also be nice.

```python
def _init_weights(self):
    raise NotImplementedError('Please implement me')
```

(If and only if all subclasses have to implement it.)

This, of course, is not my main question here.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Pass kwargs to configuration
Author: LysandreJik
Pull request: yes

**kwargs were not passed to the PreTrainedConfiguration when using `from_pretrained`

closes #3093 

Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/3147?src=pr&el=h1) Report
> Merging [#3147](https://codecov.io/gh/huggingface/transformers/pull/3147?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/7ac47bfe69f25fc7381be65870b2f4e5cdb8cb6a?src=pr&el=desc) will **increase** coverage by `<.01%`.
> The diff coverage is `100%`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/3147/graphs/tree.svg?width=650&token=9qOlN6Hb1c&height=150&src=pr)](https://codecov.io/gh/huggingface/transformers/pull/3147?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master    #3147      +/-   ##
==========================================
+ Coverage      78%   78.01%   +<.01%     
==========================================
  Files          98       98              
  Lines       16561    16569       +8     
==========================================
+ Hits        12919    12926       +7     
- Misses       3642     3643       +1
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/3147?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [src/transformers/configuration\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/3147/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9jb25maWd1cmF0aW9uX3V0aWxzLnB5) | `96.72% <100%> (+0.23%)` | :arrow_up: |
| [src/transformers/modeling\_tf\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/3147/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ190Zl91dGlscy5weQ==) | `93.41% <0%> (-0.22%)` | :arrow_down: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/3147?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/3147?src=pr&el=footer). Last update [7ac47bf...1159cff](https://codecov.io/gh/huggingface/transformers/pull/3147?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
NER pipeline usage examples
Author: jensam
Pull request: no

In the NER usage examples:
https://huggingface.co/transformers/usage.html#named-entity-recognition

Can you explain why the examples give only I- entities? For example:
('New', 'I-LOC'), ('York', 'I-LOC'), ('City', 'I-LOC')
('Hu', 'I-ORG'), ('##gging', 'I-ORG'), ('Face', 'I-ORG'), ('Inc', 'I-ORG')

Why are there no B-s? As in:
('New', 'B-LOC'), ('York', 'I-LOC'), ('City', 'I-LOC') etc

Comments:
The German and English CoNLL datasets use the IOB1 tagging scheme. `B-` is only used to separate two adjacent entities of the same type.
Finetuning FlauBERT with hugging face's Transformers : WARNING - __main__ -   Process rank: -1, device: cuda, n_gpu: 2, distributed training: False, 16-bits training: True
Author: keloemma
Pull request: no

I am trying to use the script run_glue present in the transformers library to fintenue Flaubert for french data using huggingface's transformers and I am gettinh this error :

`

03/27/2020 11:54:48 - WARNING - __main__ -   Process rank: -1, device: cuda, n_gpu: 2, distributed training: False, 16-bits training: True
Traceback (most recent call last):
  File "/home/getalp/kelodjoe/transformers/examples/run_glue.py", line 693, in <module>
    main()
  File "/home/getalp/kelodjoe/transformers/examples/run_glue.py", line 613, in main
    config_class, model_class, tokenizer_class = MODEL_CLASSES[args.model_type]
KeyError: 'flaubert'

`

Could you help me find how to solve it ?

Comments:
You're probably on an old transformers version.
T5 Summarization
Author: cformosa
Pull request: no

# 🐛 Bug
T5 summarization code in pipelines.py file gives an error.

## Information

Model I am using: T5

Language I am using the model on (English, Chinese ...): English

The problem arises when using:
pipelines.py official documentation example at line 1146
```
# use t5 in tf
summarizer = pipeline("summarization", model="t5-base", tokenizer="t5-base", framework="tf")
summarizer("Sam Shleifer writes the best docstring examples in the whole world.", min_length=5, max_length=20)
```

## To reproduce
In google colab, I did the following:
```
!pip install transformers
from transformers import pipeline

summarizer = pipeline("summarization", model="t5-base", tokenizer="t5-base", framework="tf")

summarizer("Sam Shleifer writes the best docstring examples in the whole world.", min_length=5, max_length=10)
```

And this gets the following error:
```
Your max_length is set to 200, but you input_length is only 18. You might consider decreasing max_length manually, e.g. summarizer('...', max_length=50)
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-23-0fc603a01733> in <module>()
----> 1 summarizer("Sam Shleifer writes the best docstring examples in the whole world.", min_length=5, max_length=10)

3 frames
/usr/local/lib/python3.6/dist-packages/tensorflow/python/keras/engine/base_layer.py in __call__(self, *args, **kwargs)
    797     else:
    798       raise ValueError(
--> 799           'The first argument to `Layer.call` must always be passed.')
    800 
    801     call_context = base_layer_utils.call_context()

ValueError: The first argument to `Layer.call` must always be passed.
```

     
- `transformers` version: 2.7.0
- Platform: Google Colab
- PyTorch version (GPU?): 1.4.0 GPU enabled
- Tensorflow version (GPU?): 2.2.0-rcl GPU enabled
- Using GPU in script?: 
- Using distributed or parallel set-up in script?: No


Comments:
Hi @cformosa, 

Thanks for posting this bug. This seems to be related to the new TF 2.2 release. 
Could you instead use TF2.1 for the moment:

```
!pip install transformers
!pip install tensorflow==2.1
from transformers import pipeline

summarizer = pipeline("summarization", model="t5-base", tokenizer="t5-base", framework="tf")

summarizer("Sam Shleifer writes the best docstring examples in the whole world.", min_length=5, max_length=10
```

Please let me know if you still encounter problems.
Get Warning Message: Unable to convert output to tensors format pt
Author: sharpant
Pull request: no
Labels: name: wontfix

I am running the following code: 

```
from transformers.modeling_tf_bert import TFBertForSequenceClassification
pytorch_model = TFBertForSequenceClassification.from_pretrained('./save/')

# Quickly test a few predictions - MRPC is a paraphrasing task, let's see if our model learned the task
sentence_0 = "This research was consistent with his findings."
sentence_1 = "His findings were compatible with this research."
sentence_2 = "His findings were not compatible with this research."

inputs_1 = tokenizer.encode_plus(sentence_0, sentence_1, add_special_tokens=True, return_tensors='pt')
inputs_2 = tokenizer.encode_plus(sentence_0, sentence_2, add_special_tokens=True, return_tensors='pt')
```
Get Warning Message: 

```
WARNING:transformers.tokenization_utils:Unable to convert output to tensors format pt, PyTorch or TensorFlow is not available.
WARNING:transformers.tokenization_utils:Unable to convert output to tensors format pt, PyTorch or TensorFlow is not available.
```

Then, when I run:

```
pred_1 = pytorch_model(inputs_1['input_ids'], token_type_ids=inputs_1['token_type_ids'])[0].argmax().item()
pred_2 = pytorch_model(inputs_2['input_ids'], token_type_ids=inputs_2['token_type_ids'])[0].argmax().item()

print("sentence_1 is", "a paraphrase" if pred_1 else "not a paraphrase", "of sentence_0")
print("sentence_2 is", "a paraphrase" if pred_2 else "not a paraphrase", "of sentence_0")
```

I get the error:  `AssertionError: Too many inputs.`

Comments:
It seems that you are loading a tensorflow model, which you incorrectly call pytorch_model. The reason that the function doesn't work, though, is probably because you don't have pytorch installed and only tensorflow. Convert to tensorflow tenors instead 

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.