Issue 1:
Environment info

transformers version: 4.0.1

Platform: Linux-4.15.0-132-generic-x86_64-with-glibc2.10

Python version: 3.8.5

PyTorch version (GPU?): 1.7.1+cu110 (True)

Tensorflow version (GPU?): not installed (NA)

Using GPU in script?:

Using distributed or parallel set-up in script?: no

Note also: cookiecutter dependency is not included in pip install transformers so transformers-cli env initially fails

Who can help

(T5) @patrickvonplaten, @patil-suraj, @sshleifer

When using T5ForConditionalGeneration.from_pretrained('t5-base'), I get the following warning at load:

Some weights of the model checkpoint at t5-large were not used when initializing T5ForConditionalGeneration: ['decoder.block.0.layer.1.EncDecAttention.relative_attention_bias.weight']
- This IS expected if you are initializing T5ForConditionalGeneration from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining m
odel).
- This IS NOT expected if you are initializing T5ForConditionalGeneration from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification mode
l).

If I load from a checkpoint that I create (i.e. local file), I get the same message.

But I think that all weights are, in fact, identical:

evaluation code on the model I finetune before saving AND

evaluation code on the model I finetune, save, and then reload

are identical.

This suggests that all weights are identical, since performance is identical. This contradicts the warning message.

Questions:

Are some weights actually not being loaded? If so, how could I observe identical behavior on metrics. Or is this warning wrong?

If this warning is correct, how can I force the model to fully load the model exactly as I saved it.

Is there any other difference (randomly initialized head, randomly initialized weights) between the t5 that is pretrained and the T5ForConditionalGeneration?

Issue 2:
Environment info

transformers version: 4.0.0

Platform: Linux 18.04

Python version: 3.6.9

PyTorch version (GPU?): 1.70 (True)

Tensorflow version (GPU?): N/A

Using GPU in script?: Yes

Using distributed or parallel set-up in script?: No

Who can help

@patrickvonplaten

Information

Model I am using: T5ForConditionalGeneration, specifically the "allenai/unifiedqa-t5-large" checkpoint from the model hub

The problem arises when I try to load the checkpoint following standard loading procedures under transformers==4.0.0. The same doesn't happen in version 3.5.0.

To reproduce

from transformers import AutoTokenizer, T5ForConditionalGeneration
model_name_or_path = "allenai/unifiedqa-t5-large"
tokenizer = AutoTokenizer.from_pretrained(model_name_or_path)
# Warnings in version 4.0, not in 3.5.0 an preceding ones
model = T5ForConditionalGeneration.from_pretrained(model_name_or_path)

Some weights of the model checkpoint at allenai/unifiedqa-t5-large were not used when initializing T5ForConditionalGeneration: ['decoder.blo
ck.0.layer.1.EncDecAttention.relative_attention_bias.weight']
- This IS expected if you are initializing T5ForConditionalGeneration from the checkpoint of a model trained on another task or with another
architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).
- This IS NOT expected if you are initializing T5ForConditionalGeneration from the checkpoint of a model that you expect to be exactly ident
ical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).

Expected behavior

A consistent behavior across versions, either always or never raising the warning at loading time.