noamaanMulla-03/transformers-issues huggingface dataset:
test_torch_fillmask failing on GPU
Author: sshleifer
Pull request: no
Labels: name: Help wanted

```bash
FAILED tests/test_modeling_bart.py::MBartIntegrationTests::test_enro_forward

        expected_slice = torch.tensor([9.0078, 10.1113, 14.4787], device=torch_device, dtype=model.dtype)
        result_slice = logits[0][0][:3]
>       self.assertTrue(torch.allclose(expected_slice, result_slice, atol=TOLERANCE))
E       AssertionError: False is not true

tests/test_modeling_bart.py:258: AssertionError

FAILED tests/test_pipelines.py::MonoColumnInputTestCase::test_torch_fill_mask_results

tests/test_pipelines.py:147: in _test_mono_column_pipeline
    set([o[key] for o in result]), set([o[key] for o in expect]),
E   AssertionError: Items in the first set but not the second:
E   '
My name is John
'
E   '
My name is Chris
'
E   Items in the second set but not the first:
E   '
 My name is John
'
E   '
 My name is:
'

```

cc @julien-c I'll figure it out.

Comments:
note to self: enro also fails on cpu, and rerunning on commit where it was green now fails. So change is something S3 related.


```python
tests/test_pipelines.py:148: in _test_mono_column_pipeline
    set([o[key] for o in result]), set([o[key] for o in expect]),
E   AssertionError: Items in the first set but not the second:
E   '
My name is Chris
'
E   '
My name is John
'
E   Items in the second set but not the first:
E   '
 My name is John
'
E   '
 My name is:
'
```
[mobilebert] Avoid F.tanh deprecation warning
Author: sshleifer
Pull request: yes
Labels: name: cleanup


Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/5413?src=pr&el=h1) Report
> Merging [#5413](https://codecov.io/gh/huggingface/transformers/pull/5413?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/ac611145926ff63ee6d6cbd0b28c19bacb6f7ea1&el=desc) will **increase** coverage by `0.44%`.
> The diff coverage is `100.00%`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/5413/graphs/tree.svg?width=650&height=150&src=pr&token=9qOlN6Hb1c)](https://codecov.io/gh/huggingface/transformers/pull/5413?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master    #5413      +/-   ##
==========================================
+ Coverage   77.42%   77.87%   +0.44%     
==========================================
  Files         140      140              
  Lines       24334    24334              
==========================================
+ Hits        18841    18949     +108     
+ Misses       5493     5385     -108     
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/5413?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [src/transformers/modeling\_mobilebert.py](https://codecov.io/gh/huggingface/transformers/pull/5413/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ19tb2JpbGViZXJ0LnB5) | `88.90% <100.00%> (ø)` | |
| [src/transformers/modeling\_tf\_distilbert.py](https://codecov.io/gh/huggingface/transformers/pull/5413/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ190Zl9kaXN0aWxiZXJ0LnB5) | `66.25% <0.00%> (-32.52%)` | :arrow_down: |
| [src/transformers/generation\_tf\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/5413/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9nZW5lcmF0aW9uX3RmX3V0aWxzLnB5) | `86.43% <0.00%> (+0.25%)` | :arrow_up: |
| [src/transformers/tokenization\_albert.py](https://codecov.io/gh/huggingface/transformers/pull/5413/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy90b2tlbml6YXRpb25fYWxiZXJ0LnB5) | `87.50% <0.00%> (+58.65%)` | :arrow_up: |
| [src/transformers/modeling\_tf\_electra.py](https://codecov.io/gh/huggingface/transformers/pull/5413/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ190Zl9lbGVjdHJhLnB5) | `95.38% <0.00%> (+68.46%)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/5413?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/5413?src=pr&el=footer). Last update [ac61114...0845ecd](https://codecov.io/gh/huggingface/transformers/pull/5413?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
How to merge TFDistilBertForSequenceClassification with another tf.Keras model
Author: amaiya
Pull request: no
Labels: name: wontfix

## ❓ Questions & Help

In TensorFlow 2, what is the recommended way to merge `TFDistilBertForSequenceClassification` (or any other Transformer model) with another `tf.keras` model?

In other words, I'd like to do something like this:

```
merged_out = keras.layers.concatenate([other_model.output, distilbert_model.output])
merged_out = layers.Dense(1)(merged_out)
combined_model = keras.Model([other_model.input] + distilbert_model.input, merged_out)
```
The above produces an error because `distilbert_model.output` is not accessible in the same way as vanilla tf.Keras models:
```
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-48-b15ccf9c0221> in <module>()
----> 1 merged_out = keras.layers.concatenate([other_model.output, distilbert_model.output])

/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/keras/engine/base_layer.py in output(self)
   1574     """
   1575     if not self._inbound_nodes:
-> 1576       raise AttributeError('Layer ' + self.name + ' has no inbound nodes.')
   1577     return self._get_node_attribute_at_index(0, 'output_tensors', 'output')
   1578 

AttributeError: Layer tf_distil_bert_for_sequence_classification has no inbound nodes.
```



Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


Wait, is this just the end? Am also interested in doing this

This [comment](https://github.com/huggingface/transformers/issues/4733#issuecomment-647414520) may help you?
Refactor generation sampling parameters (e.g. top k, temperature) into "Sampling" classes
Author: turtlesoupy
Pull request: yes

#4164 has a full description of the intention here. Basically, to avoid exploding `generate(...)` with more arguments, I've added one generic `Sampler` parameter that allows for arbitrary transformations of the generation probability distribution conditioned on the past. This allows users to specify custom ways of sampling (e.g. insert a specific token after a previous one, etc.) 

In the process, I've added some basic tests around these samplers; existing tests pass otherwise. 

Comments:
(Replaced merge with rebase -- see #5420)
AssertionError: Padding_idx must be within num_embeddings
Author: aranciokov
Pull request: no

### Information

Model I am using: XLM

### To reproduce

```
from pytorch_transformers import XLMModel, XLMConfig, XLMTokenizer
model_class, tokenizer_class, pretrained_weights = (XLMModel, XLMTokenizer, 'xlm-mlm-en-2048')
config = XLMConfig.from_pretrained(pretrained_weights)
xlm_model = model_class.from_pretrained(pretrained_weights, config=config)
```

**Error:** 
```
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/username/venv27/lib/python2.7/site-packages/pytorch_transformers/modeling_utils.py", line 536, in from_pretrained
    model = cls(config, *model_args, **model_kwargs)
  File "/home/username/venv27/lib/python2.7/site-packages/pytorch_transformers/modeling_xlm.py", line 545, in __init__
    self.embeddings = nn.Embedding(self.n_words, self.dim, padding_idx=self.pad_index)
  File "/home/username/venv27/lib/python2.7/site-packages/torch/nn/modules/sparse.py", line 88, in __init__
    assert padding_idx < self.num_embeddings, 'Padding_idx must be within num_embeddings'
AssertionError: Padding_idx must be within num_embeddings
```

### Expected behavior

Until recently (last I checked was 2 months ago, but since then I never updated anything), everything worked without issues and I was able to load the model with those 4 instructions easily (which is what I am expecting).

### Environment info

pytorch_transformers: 1.2.0

Python: 2.7

pytorch: 1.4.0

Comments:
I have no problem loading this in `transformers`. These files have not been updated in the past two months either; can you try doing the same in `transformers` or are you constrained by Python 2?

I'm constrained to Python 2 right now (due to a decently large codebase which I do not have enough time to upgrade to Py3 in these months), and that's why I'm using such an old version the library (which is the latest available for Py2).

Hmm, okay I think I found the source of the issue. It seems the configuration file was indeed changed, the April 24th in a non-backwards compatible way (cc @julien-c). Very sorry about this.

In order to fix this you can use this JSON as a file on your machine:

```json
{
  "architectures": [
    "XLMWithLMHeadModel"
  ],
  "emb_dim": 2048,
  "n_layers": 12,
  "n_heads": 16,
  "dropout": 0.1,
  "attention_dropout": 0.1,
  "gelu_activation": true,
  "sinusoidal_embeddings": false,
  "asm": false,
  "bos_index": 0,
  "eos_index": 1,
  "pad_index": 2,
  "unk_index": 3,
  "mask_index": 5,
  "n_langs": 1,
  "n_words": 30145
}
```

and then load it as you did, but replacing the weights name with the filepath:

```py
from pytorch_transformers import XLMModel, XLMConfig, XLMTokenizer

model_class, tokenizer_class, pretrained_weights = (XLMModel, XLMTokenizer, 'xlm-mlm-en-2048')
config = XLMConfig.from_pretrained(PATH_TO_FILE)
xlm_model = model_class.from_pretrained(pretrained_weights, config=config)
```

Everything's working now (both the four-liner and my codebase as well!). Thanks! 

Glad you could make it work!
Why T5 do not generate the whole next sentence as one of the pretrain loss?
Author: guotong1988
Pull request: no

T5 has the next-sentence-predict loss. 
And the next-sentence-predict loss is not generating the whole next sentence but generating is_next or not_next token.

Thank you very much.

Comments:
Hi @guotong1988  what do you exactly mean by not generating the whole next sentence ?

Sorry, my mistake.
Input the previous sentence and generate the next sentence, instead of input the two sentences.

I don't really understand the question here. Are you asking why T5 was not trained with the objective to generate the complete next sentence? 

Thank you for your reply.
Yes. I am asking why T5 was not pre-trained with the objective to generate the complete next sentence?

Do you think generating the whole next sentence as one of the pretrain loss will improve pretrain performance?

![image](https://user-images.githubusercontent.com/4702353/86194695-c5af5980-bb81-11ea-98d4-487355f8bb60.png)
https://github.com/google-research/text-to-text-transfer-transformer/issues/286
[fix] slow fill_mask test failure
Author: sshleifer
Pull request: yes

- New tokenizer API does not put space between `<s/>` and sentence.
- New result: "my name is John" is better than old result: "My name is"  so fine to update `expected_result`.
- This is caused by tokenizers upgrade.


Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/5406?src=pr&el=h1) Report
> Merging [#5406](https://codecov.io/gh/huggingface/transformers/pull/5406?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/c4d4e8bdbd25d9463d41de6398940329c89b7fb6&el=desc) will **decrease** coverage by `0.02%`.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/5406/graphs/tree.svg?width=650&height=150&src=pr&token=9qOlN6Hb1c)](https://codecov.io/gh/huggingface/transformers/pull/5406?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master    #5406      +/-   ##
==========================================
- Coverage   77.90%   77.87%   -0.03%     
==========================================
  Files         140      140              
  Lines       24334    24334              
==========================================
- Hits        18957    18950       -7     
- Misses       5377     5384       +7     
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/5406?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [src/transformers/pipelines.py](https://codecov.io/gh/huggingface/transformers/pull/5406/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9waXBlbGluZXMucHk=) | `75.31% <ø> (ø)` | |
| [src/transformers/modeling\_tf\_distilbert.py](https://codecov.io/gh/huggingface/transformers/pull/5406/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ190Zl9kaXN0aWxiZXJ0LnB5) | `66.25% <0.00%> (-32.52%)` | :arrow_down: |
| [src/transformers/generation\_tf\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/5406/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9nZW5lcmF0aW9uX3RmX3V0aWxzLnB5) | `86.43% <0.00%> (+0.25%)` | :arrow_up: |
| [src/transformers/file\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/5406/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9maWxlX3V0aWxzLnB5) | `79.59% <0.00%> (+0.33%)` | :arrow_up: |
| [src/transformers/modeling\_tf\_bert.py](https://codecov.io/gh/huggingface/transformers/pull/5406/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ190Zl9iZXJ0LnB5) | `98.37% <0.00%> (+25.00%)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/5406?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/5406?src=pr&el=footer). Last update [c4d4e8b...bd7b994](https://codecov.io/gh/huggingface/transformers/pull/5406?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


lgtm
RuntimeError: Error(s) in loading state_dict for BertForTokenClassification
Author: pz325
Pull request: no
Labels: name: wontfix

Since about 1600 BST, 24 Apr 2020, loading ner pipeline gives RuntimeError:
```
from transformers import pipeline
nlp = pipeline("ner", ignore_labels=[])
```

> Traceback (most recent call last):
  File "test.py", line 2, in <module>
    nlp = pipeline("ner", ignore_labels=[])
  File "/Users/xxx/Github/xxx/xxx/.venv/lib/python3.6/site-packages/transformers/pipelines.py", line 1091, in pipeline
    model = model_class.from_pretrained(model, config=config, **model_kwargs)
  File "/Users/xxx/Github/xxx/xxx/.venv/lib/python3.6/site-packages/transformers/modeling_auto.py", line 1086, in from_pretrained
    return model_class.from_pretrained(pretrained_model_name_or_path, *model_args, config=config, **kwargs)
  File "/Users/xxx/Github/xxx/xxx/.venv/lib/python3.6/site-packages/transformers/modeling_utils.py", line 558, in from_pretrained
    model.__class__.__name__, "\n\t".join(error_msgs)
RuntimeError: Error(s) in loading state_dict for BertForTokenClassification:
        size mismatch for classifier.weight: copying a param with shape torch.Size([9, 1024]) from checkpoint, the shape in current model is torch.Size([2, 1024]).
        size mismatch for classifier.bias: copying a param with shape torch.Size([9]) from checkpoint, the shape in current model is torch.Size([2]).

- `transformers` version: 2.5.1
- Platform: macos 10.14.6
- Python version:  3.6.5
- PyTorch version (GPU?): 
- Tensorflow version (GPU?): 2.0.0
- Using GPU in script?: 
- Using distributed or parallel set-up in script?:


Comments:
An update: after upgrading to transformers 2.8.0, the reported issue is gone. 

Nevertheless, could you share an explanation please? 

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
BertForSequenceClassification is not optimum
Author: RodSernaPerez
Pull request: no
Labels: name: wontfix

In the documentation about TFBertModel it says:

```
hidden_states (tuple(tf.Tensor), optional, returned when config.output_hidden_states=True):
tuple of tf.Tensor (one for the output of the embeddings + one for the output of each layer) of shape (batch_size, sequence_length, hidden_size).

```
Hidden-states of the model at the output of each layer plus the initial embedding outputs.

But then in the code of BertForSequenceClassification:

```
        outputs = self.bert(inputs, **kwargs)

        pooled_output = outputs[1]

        pooled_output = self.dropout(pooled_output, training=kwargs.get("training", False))
        logits = self.classifier(pooled_output)

        outputs = (logits,) + outputs[2:]  # add hidden states and attention if they are here

        return outputs  # logits, (hidden_states), (attentions)
```
Wouldn't give better results using `pooled_output = outputs[0][:,0]` ?

Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Pretrain From Scratch using Google TPU
Author: muhammadfahid51
Pull request: no
Labels: name: wontfix

@julien-c @patrickvonplaten 
I want to pretrain a model from scratch by utilizing Google Cloud TPU offered in kaggle. I can train the model without TPU but I want to train it on a TPU. Any help will be much appreciated.

Also what options do I have, if there is no  straight forward approach ?

Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
How can i finetune an encode-decoder combination?
Author: Palipoor
Pull request: no
Labels: name: wontfix

Hi, I want to finetune an encode-decoder model to train on a parallel dataset(something like translation) and I'm not sure what should I do. I read [this](https://medium.com/huggingface/encoder-decoders-in-transformers-a-hybrid-pre-trained-architecture-for-seq2seq-af4d7bf14bb8) blog post but It didn't help. It doesn't really matter for me which encode or decoder I choose, I just need the power of a pre-trained transformer. 

Comments:
Well I found the summarization example, I'm going to read it and I'll close this issue if it helps. Would still be happy if you can help.

@Palipoor Can you please share the summarization example. I am looking for one.

> @Palipoor Can you please share the summarization example. I am looking for one.

Here it is:
https://github.com/huggingface/transformers/tree/master/examples/summarization

Although it didn't help me and I'm trying to finetune a T5.


This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
How to do parameter sharing between two BERT models
Author: xiepuzhao
Pull request: no
Labels: name: wontfix

# ❓ Questions & Help

<!-- The GitHub issue tracker is primarly intended for bugs, feature requests,
     new models and benchmarks, and migration questions. For all other questions,
     we direct you to Stack Overflow (SO) where a whole community of PyTorch and
     Tensorflow enthusiast can help you out. Make sure to tag your question with the
     right deep learning framework as well as the huggingface-transformers tag: 
     https://stackoverflow.com/questions/tagged/huggingface-transformers 
     
     If your question wasn't answered after a period of time on Stack Overflow, you
     can always open a question on GitHub. You should then link to the SO question 
     that you posted.
     -->

## Details
<!-- Description of your issue -->

<!-- You should first ask your question on SO, and only if
     you didn't get an answer ask it here on GitHub. -->
**A link to original question on Stack Overflow**: 

Comments:
You could do it very simply by passing the reference around:

```py
from transformers import BertModel

model = BertModel.from_pretrained("bert-base-cased")
model2 = BertModel.from_pretrained("bert-base-cased")

model2.embeddings = model.embeddings

print(model2.embeddings.word_embeddings.weight)

model.embeddings.word_embeddings.weight = torch.nn.Parameter(torch.zeros_like(model.embeddings.word_embeddings.weight))

print(model2.embeddings.word_embeddings.weight)
```

which outputs the result (note that I'm updating the `model.embeddings` and printing the `model2.embeddings`):

```py
Parameter containing:
tensor([[-0.0005, -0.0416,  0.0131,  ..., -0.0039, -0.0335,  0.0150],
        [ 0.0169, -0.0311,  0.0042,  ..., -0.0147, -0.0356, -0.0036],
        [-0.0006, -0.0267,  0.0080,  ..., -0.0100, -0.0331, -0.0165],
        ...,
        [-0.0064,  0.0166, -0.0204,  ..., -0.0418, -0.0492,  0.0042],
        [-0.0048, -0.0027, -0.0290,  ..., -0.0512,  0.0045, -0.0118],
        [ 0.0313, -0.0297, -0.0230,  ..., -0.0145, -0.0525,  0.0284]],
       requires_grad=True)
Parameter containing:
tensor([[0., 0., 0.,  ..., 0., 0., 0.],
        [0., 0., 0.,  ..., 0., 0., 0.],
        [0., 0., 0.,  ..., 0., 0., 0.],
        ...,
        [0., 0., 0.,  ..., 0., 0., 0.],
        [0., 0., 0.,  ..., 0., 0., 0.],
        [0., 0., 0.,  ..., 0., 0., 0.]], requires_grad=True)
```

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
[proposal] Move tests/utils.py to src/transformers/testing_utils.py so that examples can import
Author: sshleifer
Pull request: no
Labels: name: cleanup


for both groups of tests, the import would be
```python
from transformers.testing_utils import slow
```

Motivation: I was about to rewrite the @slow decorator today and felt that this was cleaner.

Any objections? @julien-c @LysandreJik @thomwolf @sgugger @patrickvonplaten @mfuntowicz 
@anyoneelseimforgetting


Comments:
I think that would be cool. Since we're testing the examples, it makes sense to not duplicate the exact same code.

Sounds good to me
Can't install transformers from sources using poetry
Author: simonepri
Pull request: no
Labels: name: wontfix

# 🐛 Bug

When installing transformers from sources using poetry

```bash
poetry add git+https://github.com/huggingface/transformers.git
```

The following exception is thrown:

```bash
[InvalidRequirement]
Invalid requirement, parse error at "'extra =='"
```

This is a problem on poetry side. The issue here is just to cross-link the problem.

Ref: https://github.com/python-poetry/poetry/issues/2326



Comments:
In the meantime as a workaround, you can do the following:

1) Fork huggingface/transformers
2) On your fork remove this line https://github.com/huggingface/transformers/blob/97a375484c618496691982f62518130f294bb9a8/setup.py#L79
3) Install your fork with poetry

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.