UnimplementedError: The Conv2D op currently does not support grouped convolutions on the CPU.
Author: innat
Pull request: no
Labels: name: bug

# Info
```
Coalb TPU v2
Kaggle TPU v3
TensorFlow: 2.4.1
Transformer: 4.22.0.dev0
```

# Who can help?

@Rocketknight1 @NielsRogge @sgugger @amyeroberts

# Information

- [ ] The official example scripts
- [X] My own modified scripts

# Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

# Reproduction

Please, get the file form [HERE](https://drive.google.com/file/d/1qAo4PsZ8DekZqFzF5Oq2E3nkJ3aJWFuM/view?usp=sharing). A notebook scripts, just plug-n-play.

**What to do**

1. Run the script in Colab with TPU.
2. Run the script in Kaggle with TPU. 

You may not need to change anything, just through the file to these platform and run all.

# Expected behavior

**What was I doing**

With the given script above, I was trying to run a vision transformer model on Kaggle TPU (with TF 2.4.1 by default). And I got

```python
      2     prime_input = tf.keras.Input(shape=(*IMAGE_SIZE, 3))
      3     mode_inputs = tf.keras.layers.Permute(dims=(3, 1, 2))(prime_input)
----> 4     backbone = TFConvNextModel.from_pretrained("facebook/convnext-tiny-224")
      5     backbone.trainable = False
....
    171     def call(self, hidden_states, training=False):
    172         input = hidden_states
--> 173         x = self.dwconv(hidden_states)
    174         x = self.layernorm(x)
    175         x = self.pwconv1(x)
```
> UnimplementedError: The Conv2D op currently does not support grouped convolutions on the CPU. A grouped convolution was attempted to be run because the input depth of 96 does not match the filter input depth of 1

A known tf issue, discussed also [here](https://github.com/tensorflow/tensorflow/issues/29005). But this issue didn't appear when I ran the same script on Colab TPU (with `tf 2.4.1`) system. The model build successfully. 

As I am currently using transformer on kaggle platform, I need to make it work. The given script above is just about model construction code. Any pointer what's going on here? 

Please note again, Kaggle TPU v3 and Colab TPU v2. Not sure if it's something to do with this.

Comments:
cc @gante 

Hey @innat 👋 That exception doesn't come from `transformers`, but from TensorFlow itself. As you mentioned, using a newer TF version does not result in an error. 

I'm afraid we won't be able to help you here :)

@gante 
Sorry I think I didn't explain well. Here is short and exact summary. 

> I ran transformer on Colab TPU and Kaggle TPU with TensorFlow 2.4.1. The script that I shared above works in Colab TPU but throws error in Kaggle TPU. 

Please let me know if you had trouble to run the script.

@gante Here are the prepared notebook. It would be easy to test.

- [kaggle-tpu](https://www.kaggle.com/code/ipythonx/transformer-issue-github-18792/)
- [colab-tpu](https://drive.google.com/file/d/1qAo4PsZ8DekZqFzF5Oq2E3nkJ3aJWFuM/view?usp=sharing)

Grouped convolution on CPU were added in Tensorflow 2.5, so 2.4 won't work. That's why colab with Tensorflow 2.9 isn't affected. Have you tried updating Tensorflow on kaggle with `!pip install -U tensorflow`?

I set accelerator **TPU** to both colab and kaggle environments. I'm using `TF 2.4.1` on both environment. With this set up, convn-next model successfully built on colab tpu (with `tf 2.4.1`) but didn't work on kaggle tpu (with `tf 2.4.1`). 

Please note, I'm **NOT** using `TF 2.9.1`, either in colab or kaggle environments. Have you run the the code that I shared above. I think it's straightforward to get on the same page.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

> This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.
> 
> Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

The last comemnt came from the original OP. This bot message doesn't make sense. Instead of such alert, it should tag `stat:awaiting:transformers`.

Hi @innat -- as I've mentioned above this is an issue for the Kaggle and/or the TensorFlow team, there is nothing the `transformers` team can do. 

We don't have the power to go back in time and add code to a repository that isn't ours, nor to update Kaggle's TPU runtimes.