Loading a Pytorch model from a Tensorflow saved model doesn't work
Author: arjunaskykok
Pull request: no
Labels: name: TensorFlow

name: bug

### System Info

- `transformers` version: 4.51.3
- Platform: Linux-6.1.123+-x86_64-with-glibc2.35
- Python version: 3.11.12
- Huggingface_hub version: 0.30.2
- Safetensors version: 0.5.3
- Accelerate version: 1.5.2
- Accelerate config: 	not found
- DeepSpeed version: not installed
- PyTorch version (GPU?): 2.6.0+cu124 (True)
- Tensorflow version (GPU?): 2.18.0 (True)
- Flax version (CPU?/GPU?/TPU?): 0.10.5 (gpu)
- Jax version: 0.5.2
- JaxLib version: 0.5.1
- Using distributed or parallel set-up in script?: No
- Using GPU in script?: No
- GPU type: Tesla T4


### Who can help?

@gante and @Rocketknight1

### Information

- [x] The official example scripts
- [ ] My own modified scripts

### Tasks

- [x] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

I got the error from [the Quickstart notebook](https://github.com/huggingface/notebooks/blob/main/transformers_doc/en/quicktour.ipynb).

To make it easier for reproduction, here's the sample script to reproduce the error:

```python
from transformers import TFAutoModelForSequenceClassification

#model_name = "nlptown/bert-base-multilingual-uncased-sentiment"
model_name = "bert-base-cased"
tf_model = TFAutoModelForSequenceClassification.from_pretrained(model_name)

from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained(model_name)

tf_save_directory = "./tf_save_pretrained"
tokenizer.save_pretrained(tf_save_directory)
tf_model.save_pretrained(tf_save_directory)

from transformers import AutoModelForSequenceClassification

pt_model = AutoModelForSequenceClassification.from_pretrained(tf_save_directory, from_tf=True)
```

You would get this error:

```
/usr/local/lib/python3.11/dist-packages/torch/nn/modules/module.py:2397: UserWarning: for bert.embeddings.word_embeddings.weight: copying from a non-meta parameter in the checkpoint to a meta parameter in the current model, which is a no-op. (Did you mean to pass `assign=True` to assign items in the state dictionary to their corresponding key in the module instead of copying them in place?)
  warnings.warn(
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
[<ipython-input-7-95e44b6f2430>](https://localhost:8080/#) in <cell line: 0>()
----> 1 pt_model = AutoModelForSequenceClassification.from_pretrained(tf_save_directory, from_tf=True)

8 frames
[/usr/local/lib/python3.11/dist-packages/torch/nn/modules/module.py](https://localhost:8080/#) in load_state_dict(self, state_dict, strict, assign)
   2579 
   2580         if len(error_msgs) > 0:
-> 2581             raise RuntimeError(
   2582                 "Error(s) in loading state_dict for {}:\n\t{}".format(
   2583                     self.__class__.__name__, "\n\t".join(error_msgs)

RuntimeError: Error(s) in loading state_dict for BertForSequenceClassification:
	size mismatch for bert.embeddings.position_embeddings.weight: copying a param with shape torch.Size([28996, 768]) from checkpoint, the shape in current model is torch.Size([512, 768]).
	size mismatch for bert.embeddings.token_type_embeddings.weight: copying a param with shape torch.Size([28996, 768]) from checkpoint, the shape in current model is torch.Size([2, 768]).
	size mismatch for bert.embeddings.LayerNorm.weight: copying a param with shape torch.Size([28996, 768]) from checkpoint, the shape in current model is torch.Size([768]).
	size mismatch for bert.embeddings.LayerNorm.bias: copying a param with shape torch.Size([28996, 768]) from checkpoint, the shape in current model is torch.Size([768]).
	size mismatch for bert.encoder.layer.0.attention.self.query.weight: copying a param with shape torch.Size([28996, 768]) from checkpoint, the shape in current model is torch.Size([768, 768]).
	size mismatch for bert.encoder.layer.0.attention.self.query.bias: copying a param with shape torch.Size([28996, 768]) from checkpoint, the shape in current model is torch.Size([768]).
	size mismatch for bert.encoder.layer.0.attention.self.key.weight: copying a param with shape torch.Size([28996, 768]) from checkpoint, the shape in current model is torch.Size([768, 768]).
	size mismatch for bert.encoder.layer.0.attention.self.key.bias: copying a param with shape torch.Size([28996, 768]) from checkpoint, the shape in current model is torch.Size([768]).
...
size mismatch for bert.encoder.layer.11.output.dense.weight: copying a param with shape torch.Size([28996, 768]) from checkpoint, the shape in current model is torch.Size([768, 3072]).
	size mismatch for bert.encoder.layer.11.output.dense.bias: copying a param with shape torch.Size([28996, 768]) from checkpoint, the shape in current model is torch.Size([768]).
	size mismatch for bert.encoder.layer.11.output.LayerNorm.weight: copying a param with shape torch.Size([28996, 768]) from checkpoint, the shape in current model is torch.Size([768]).
	size mismatch for bert.encoder.layer.11.output.LayerNorm.bias: copying a param with shape torch.Size([28996, 768]) from checkpoint, the shape in current model is torch.Size([768]).
	size mismatch for bert.pooler.dense.weight: copying a param with shape torch.Size([28996, 768]) from checkpoint, the shape in current model is torch.Size([768, 768]).
	size mismatch for bert.pooler.dense.bias: copying a param with shape torch.Size([28996, 768]) from checkpoint, the shape in current model is torch.Size([768]).
	size mismatch for classifier.weight: copying a param with shape torch.Size([28996, 768]) from checkpoint, the shape in current model is torch.Size([2, 768]).
	size mismatch for classifier.bias: copying a param with shape torch.Size([28996, 768]) from checkpoint, the shape in current model is torch.Size([2]).
```



### Expected behavior

I should be able to get the Pytorch model from the Tensorflow saved model.

Loading up the Tensorflow model from the Pytorch saved model **works** (the other way around).

Comments:
"""It's a helpful, minimal reproducible example that supports the ongoing discussion."""



from transformers import TFAutoModelForSequenceClassification, AutoTokenizer, AutoModelForSequenceClassification

model_name = "bert-base-cased"

# Load TensorFlow model with a classification head
tf_model = TFAutoModelForSequenceClassification.from_pretrained(model_name, num_labels=2)

# Load tokenizer
tokenizer = AutoTokenizer.from_pretrained(model_name)

# Save TensorFlow model
tf_model.save_pretrained("./tf_model")

# Try loading as PyTorch model (this fails)
pt_model = AutoModelForSequenceClassification.from_pretrained("./tf_model", from_tf=True)  # Fails with shape mismatch


@arjunaskykok you're using `tensorflow==2.18.0`, and our library only supports `tensorflow>2.9,<2.16`. Could you try again with a compatible tensorflow version? :)

@gante The problem persists.

![Image](https://github.com/user-attachments/assets/fdc420e4-3791-4a4a-8bc4-e60d93bc6991)

@arjunaskykok thank you for trying it out 👍 

I have confirmed that the following script works if we install `transformers==4.50.0`. TF support with transformers is currently in maintenance mode, so it may take a while to fix the problem on `main` :) 

(cc @Rocketknight1 FYI in can you see similar TF loading issues)

```py
from transformers import TFAutoModelForSequenceClassification, AutoModelForSequenceClassification

model_name = "bert-base-cased"
tf_model = TFAutoModelForSequenceClassification.from_pretrained(model_name)

tf_save_directory = "/tmp/tf_save_pretrained"
tf_model.save_pretrained(tf_save_directory)

pt_model = AutoModelForSequenceClassification.from_pretrained(tf_save_directory, from_tf=True)
```

I think the bug is located inside `from_pretrained` method. I tried to cross-load-save with current version and 4.50. The 4.50 version can load the model that saved with the current version. But the other way doesn't work.

The regression starts at 4.51.0. But the error message is different.

![Image](https://github.com/user-attachments/assets/e8a10846-bdf6-4fa7-a615-67ea2b4ec0fa)



This is the commit that broke the stuff for the first time:

https://github.com/huggingface/transformers/commit/f304318f5ffa530a4948aed552d3405708163b52

The error message is different from the posted bug. But it's a start.

Why did it break?

![Image](https://github.com/user-attachments/assets/f460c6cc-d781-4237-8de9-d76105172f54)

As can be seen in the previous code, we import `init_empty_weights` in the `is_accelerate_available` function.

![Image](https://github.com/user-attachments/assets/a0a492fd-854b-4bab-906c-838e61b1802d)

We used to execute `is_accelerate_available` before accessing `init_empty_weights`. But after the commit, we don't have the guard anymore. The guard, `low_cpu_mem_usage`, has been removed.

So, the posted bug is being introduced in the version `4.51.1`.

To summarize:

- Version `4.50.3` gives no bug.
- Version `5.51.0` gives "`init_empty_weights` is not defined" bug.
- Version `5.51.1` gives "Error(s) in loading `state_dict`" bug.

On the second run, the version `5.51.0` gives this error:
```
    from .integrations.accelerate import find_tied_parameters, init_empty_weights
ModuleNotFoundError: No module named 'transformers.integrations.accelerate'
```

Previously I probably mixed up a lot of things.

Okay, I can confirm this commit ("Remove low_cpu_mem_usage and _fast_init") introduces the posted bug ("Error(s) in loading state_dict"):

https://github.com/huggingface/transformers/commit/f304318f5ffa530a4948aed552d3405708163b52

I patched the code with the fix for init empty weights bug to make the original bug appears:

https://github.com/huggingface/transformers/commit/4f536ba0ae66d3caf37072dcd67a5cbf7a2cad09

So, we can focus on the commit. Why does it introduce the bug?

I found the culprit! I'll prepare the PR.