TF implementation of RegNets
Author: ariG23498
Pull request: yes

In this PR in which we (/w @sayakpaul) are proting the RegNets model into TensorFlow.


Comments:
_The documentation is not available anymore as the PR was closed or merged._

@amyeroberts 

If we run the following:

```
from PIL import Image
import numpy as np

from src.transformers.models.regnet.modeling_tf_regnet import (
    TFRegNetForImageClassification
)
from transformers import AutoFeatureExtractor

def prepare_img():
    image = Image.open("./tests/fixtures/tests_samples/COCO/000000039769.png")
    return image

feature_extractor = AutoFeatureExtractor.from_pretrained("facebook/regnet-y-040")
model = TFRegNetForImageClassification.from_pretrained("facebook/regnet-y-040", from_pt=True)

image = prepare_img()
inputs = feature_extractor(images=image, return_tensors="tf") 
outputs = model(**inputs, training=False)

print(outputs.logits.shape)

expected_slice = np.array([-0.4180, -1.5051, -3.4836])

np.testing.assert_allclose(outputs.logits[0, :3].numpy(), expected_slice, atol=1e-4)
```

First, it complains the `moving_mean` and `moving_variance` params are not loaded properly.

We tested your solution in https://github.com/huggingface/transformers/pull/17571. With that, we're running into mismatches of `num_batches_tracked` and even `moving_mean`. It also complains about some of the mismatches stemming from the `shortcut` layer which wasn't the case for the earlier setup.

Do you have any thoughts?  

Hi @sayakpaul 

Could you give a bit more information about the mismatches i.e. the printouts you're currently getting?

Regarding `num_batches_tracked`, I don't believe this parameter will ever be cross-loaded into a `tf.keras.layers.BatchNormalization` layer as there isn't an equivalent parameter. This is only important if the corresponding PyTorch batch norm layer doesn't have its momentum set c.f. [param updates](https://github.com/pytorch/pytorch/blob/67badf0d5cefeb0d39767609e78aa5ff668a262e/torch/nn/modules/batchnorm.py#L149), which you'll need to verify for this model. I suggest looking at the implementations of both the [TF](https://github.com/keras-team/keras/blob/07e13740fd181fc3ddec7d9a594d8a08666645f6/keras/layers/normalization/batch_normalization.py#L1107-L1249) and PyTorch layer to see when/if these differences are important. If the parameter is necessary, then I think one approach might be subclassing to build a new layer and include the parameter as a registered weight + any necessary logic to use it, but I'm not sure at the moment. 

I tried debugging this today but no luck yet. But here's some information for all of us to navigate this through:

* Amending [`src/transformers/modeling_tf_pytorch_utils.py`](https://github.com/huggingface/transformers/pull/17554/files#diff-67993ece845388c9a9a1f342f4c82a2ed7f790f454134c39c3a63146616ea37a) (following https://github.com/huggingface/transformers/pull/17571) resulted in this: https://pastebin.com/0CZJmvzh.
* `num_batches_tracked` is likely not needed, I don't suspect that to be a trained parameter anyway. However, happy to stand corrected. 
* But what is surprising is even after incorporating the changes from https://github.com/huggingface/transformers/pull/17571 there's a complaint about `moving_mean` and `moving_variance`. 
* There's also a complaint about `convolution` params.

All these mismatches seem to be stemming from the `layers.0` of RegNet stages. Mismatches stemming from other `layers` (`layers.2` for example) are related to `num_batches_tracked`. 

The test used to gather this information is the same one as mentioned in https://github.com/huggingface/transformers/pull/17554#issuecomment-1147700055. 

@amyeroberts 

@sayakpaul Thanks for your detailed update. Comments below:

1. OK - thanks for posting that it really helps!

2. `num_batches_tracked` isn't trainable, but it is updated during training. As I mentioned above, if the layer has `momentum` set (it's not `None`) then you can ignore it. However, if `momentum` isn't set, then the layer uses `num_batches_tracked` to update the `running_mean` and `running_var` calculations, which are used during evaluation to normalize the batch. You can quickly check if the momentum is set for the batchnorm layers running something like `all([x.momentum is not None for x in model.modules() if isinstance(x, nn.BatchNorm2d)])`.  

3. Looking at the printout you pasted above, it says `All the weights of TFRegNetForImageClassification were initialized from the PyTorch model.`. If this is the case, and some of the PyTorch weights weren't used, it makes me think some layers might be missing in your implementation. I would look at the two architectures and see if they differ anywhere. 

@amyeroberts a quick update:

* `momentum` is actually not set. This is why we need to also retrieve `num_batches_tracked` too. We need to figure out a way to factor it in to use with `layers.BatchNormalization` in TensorFlow.
* The TF model has a fewer number of params than the PT model so we'll look into why this is the case. One immediate reason would be the absence of `num_batches_tracked`. But that contributes a very small difference. We currently have 629440 fewer parameters in the TF model than the PT one. 

@sayakpaul Thanks for the update! 

* OK, this makes things a bit more difficult. Let me know if you want any help for this step. It's something that will likely need to be done in other PT -> TF ports so definitely valuable to the community if you added this!

* It might be easier to print out the weight names instead of comparing number of parameters. The porting code works on the names, and so seeing where the two models differ can really help pinpoint what's happening. What I typically do is use the porting code to convert the tensorflow weight names and compare the two sets. For this model, it would look something like: 
```
from transformers import RegNetForImageClassification
# import directly once __init__ files updated
from transformers.models.regnet.modeling_tf_regnet import TFRegNetForImageClassification 
from transformers.modeling_tf_pytorch_utils import convert_tf_weight_name_to_pt_weight_name

checkpoint = "facebook/regnet-y-040"
tf_model = TFRegNetForImageClassification.from_pretrained(checkpoint, from_pt=True)
pt_model = RegNetForImageClassification.from_pretrained(checkpoint)

tf_model_weights = set([convert_tf_weight_name_to_pt_weight_name(x.name)[0] for x in tf_model.trainable_variables])
pt_model_weights = set(pt_model.state_dict().keys())

print(tf_model_weights - pt_model_weights)
print(pt_model_weights - tf_model_weights)
```

Thanks for the suggestions. Will try them out and update.

@amyeroberts 

I had to do a few minor modifications to your snippet in https://github.com/huggingface/transformers/pull/17554#issuecomment-1150933208:

```
tf_model_weights = set(
    [
        convert_tf_weight_name_to_pt_weight_name(x.name)[0]
        for x in tf_model.trainable_variables + tf_model.non_trainable_variables
    ]
)
pt_model_weights = set(pt_model.state_dict().keys())
tf_model_weights_new = set()

for name in tf_model_weights:
    if "moving_mean" in name:
        name = name.replace("moving_mean", "running_mean")
    elif "moving_variance" in name:
        name = name.replace("moving_variance", "running_var")
    tf_model_weights_new.add(name)


print(f"Differences in the TF model and PT model: {tf_model_weights_new - pt_model_weights}")
print(f"Differences in the PT model and TF model: {pt_model_weights - tf_model_weights_new}")
print(f"Total weights differing: {len(pt_model_weights - tf_model_weights_new)}")
```

`convert_tf_weight_name_to_pt_weight_name()` doesn't change the `moving_mean` and `moving_variance` to `running_mean` and `running_var` respectively. Instead, currently, it's handled [here](https://github.com/ariG23498/transformers/blob/aritra-regnets/src/transformers/modeling_tf_pytorch_utils.py#L160-#L172) so that [this query](https://github.com/ariG23498/transformers/blob/aritra-regnets/src/transformers/modeling_tf_pytorch_utils.py#L205) is successful. 

With this change, the result of `pt_model_weights - tf_model_weights_new` is exactly matching with the complaint: 

```
Some weights of the PyTorch model were not used when initializing the TF 2.0 model TFRegNetForImageClassification ...
```

(Full output [here](https://pastebin.com/cg10ET1c)). 

I have gone over the `modeling_tf_regnet.py` script a couple of times but I don't yet know what I can do here. Let me know what you usually do when you have these differences. 

Also an oversight on my end in reporting `momentum` in https://github.com/huggingface/transformers/pull/17554#issuecomment-1150851986.

`all([x.momentum is not None for x in model.modules() if isinstance(x, nn.BatchNorm2d)])` actually gives `True` which means it's okay to ignore `num_batches_tracked`. 

@amyeroberts we were able to rectify the model implementation and make it work. The integration test (mentioned in https://github.com/huggingface/transformers/pull/17554#issuecomment-1147700055) is passing now.

The tests, however, are failing for a weird reason:

```
Parameter config in `TFRegNetModel(config)` should be an instance of class `PretrainedConfig`. To create a model from a pretrained model use `model = TFRegNetModel.from_pretrained(PRETRAINED_MODEL_NAME)`
```

Weird because we tested a couple of things in isolation:

```py
from transformers import RegNetConfig

config_class = RegNetConfig()

print(f"RegNet Config class type: {type(config_class)}.")
print(f"RegNet Config is an instance of PretrainedConfig: {isinstance(config_class, PretrainedConfig)}")
```

The final print statement gives `True`. But when we do the following:

```py
from src.transformers.models.regnet.modeling_tf_regnet import TFRegNetForImageClassification, TFRegNetModel

class_from_config = TFRegNetModel(config_class)
print("Model class from config was initialized.")
```

it complains:

```
Parameter config in `TFRegNetModel(config)` should be an instance of class `PretrainedConfig`. To create a model from a pretrained model use `model = TFRegNetModel.from_pretrained(PRETRAINED_MODEL_NAME)`
```

Do you have any suggestions for this?

@sgugger @Rocketknight1 the PR is now ready for review. 

This particular model actually has the largest vision model checkpoint available to date: https://huggingface.co/facebook/regnet-y-10b-seer. It's still in PyTorch and the corresponding model makes use of the `low_cpu_usage` argument. 

I had a chat with @Rocketknight1  a few days back on the possibility of supporting this checkpoint in TensorFlow too. This will require tweaks and they will be contributed in a separate PR. 

@ydshieh I thought I could use your help here. There's something really weird happening here. 

If I omit the `image_size` argument from the [`RegNetConfig`](https://github.com/ariG23498/transformers/blob/aritra-regnets/src/transformers/models/regnet/configuration_regnet.py#L76) the cross-testing is failing (full [stack-trace](https://pastebin.com/Us6BgKvh)).

```
RUN_SLOW=1 RUN_PT_TF_CROSS_TESTS=1 python -m pytest tests/models/regnet/test_modeling_tf_regnet.py
```

But if I keep the argument, it runs successfully. The only use of `image_size` is [here](https://github.com/ariG23498/transformers/blob/aritra-regnets/src/transformers/models/regnet/modeling_tf_regnet.py#L425). 

In any case, the PyTorch cross-test of the same model is failing ([full stack-trace with `image_size` set, trace without `image_size` set](https://pastebin.com/LjmPjqpZ)):

```
 RUN_SLOW=1 RUN_PT_TF_CROSS_TESTS=1 python -m pytest tests/models/regnet/test_modeling_regnet.py
```

Have you got any suggestions? 

Cc: @Rocketknight1 


Hey @sayakpaul 

By `I omit the image_size argument from the` [RegNetConfig](https://github.com/ariG23498/transformers/blob/aritra-regnets/src/transformers/models/regnet/configuration_regnet.py#L76), could you specify where you call config without passing `image_size `. I guess you mean somewhere in the TF Reg test file, but which line exactly 🙏 ?

`dummy_inputs` is also used in https://github.com/huggingface/transformers/blob/3c7e56fbb11f401de2528c1dcf0e282febc031cd/src/transformers/modeling_tf_utils.py#L1975

to build the network, so the weights (with random values) will present in order to load the real weights. My best guess is that without specifying somewhere, the model is initialized to handle size 224 images, and it causes some shape inconsistency issue for your test. For more detailed investigation, I need to know where you don't specify this argument. But probably you are able to figure this out with this info. maybe?

BTW, why you need `pastebin` to post the traceback? A link to the CircleCI job run page is ok, no?

@ydshieh thanks for your inputs. 

So, what I meant is I just comment [this line](https://github.com/ariG23498/transformers/blob/aritra-regnets/src/transformers/models/regnet/configuration_regnet.py#L76) and [this line](https://github.com/ariG23498/transformers/blob/aritra-regnets/src/transformers/models/regnet/configuration_regnet.py#L89) and just pass a hardcoded value (224) [here](https://github.com/ariG23498/transformers/blob/aritra-regnets/src/transformers/models/regnet/modeling_tf_regnet.py#L425) (`3, self.config.num_channels, 224, 224`). Is this better to understand now? 

So, if `image_size` specified in the config (like it is currently), the cross-test with TF (`test_modeling_tf_regnet.py`) passes successfully but not with PT (i.e., `test_modeling_regnet.py`). 


> BTW, why you need `pastebin` to post the traceback? A link to the CircleCI job run page is ok, no?

The CI trace seemed clunky so I ran the individual test to keep the outputs cleaner. 

The TF implementation of `TFAdaptiveAvgPool1D` (used for `TFAdaptiveAvgPool2D`) is not capable to handle any image size (once being initialized with an input), as its `build` method contains `self.map = tf.constant(sparse_map)`, which determines the input shape it can handle in the subsequent calls.

I didn't check how PyTorch implements `nn.AdaptiveAvgPool2d((1, 1))`, but I think it is possible not to build `self.map`, but instead to prepare the necessary matrix dynamically in the `call` method.

Regarding the test, the situation is similar: the method `dummy_inputs` is called in `from_pretrained` method. While hard-coded with `224`, the model can't run with other image size anymore. Even if you add `image_size` argument in `RegNetConfig`, once a 
`TFRegNetModel` is run with an input, it can't run with other input shape anymore. This contradicts to the PyTorch implementation.

Maybe you can figure out a more robust way for TF?

Here is the code snippet to make things more concrete:
```python
import torch
import tensorflow as tf
import math
from typing import List
import numpy as np


# Copied from:
# https://gist.github.com/Rocketknight1/43abbe6e73f1008e6e459486e01e0ceb
class TFAdaptiveAvgPool1D(tf.keras.layers.Layer):
    def __init__(self, output_dim, mode="dense", **kwargs):
        super().__init__(**kwargs)
        self.output_dim = output_dim
        self.mode = mode
        self.map = None

    def build(self, input_shape):
        super().build(input_shape)
        """We pre-compute the sparse matrix for the build() step once. The below code comes
        from https://stackoverflow.com/questions/53841509/how-does-adaptive-pooling-in-pytorch-work/63603993#63603993."""

        def get_kernels(ind, outd) -> List:
            """Returns a List [(kernel_offset_start,kernel_length)] defining all the pooling kernels for a 1-D adaptive
            pooling layer that takes an input of dimension `ind` and yields an output of dimension `outd`"""

            def start_index(a, b, c):
                return math.floor((float(a) * float(c)) / b)

            def end_index(a, b, c):
                return math.ceil((float(a + 1) * float(c)) / b)

            results = []
            for ow in range(outd):
                start = start_index(ow, outd, ind)
                end = end_index(ow, outd, ind)
                sz = end - start
                results.append((start, sz))
            return results

        in_dim = int(input_shape[-1])
        kernels = get_kernels(in_dim, self.output_dim)
        sparse_map = np.zeros((in_dim, self.output_dim), dtype=np.float32)
        for i, kernel in enumerate(kernels):
            sparse_map[kernel[0] : kernel[0] + kernel[1], i] = 1 / kernel[1]
        if self.mode == "dense":
            self.map = tf.constant(sparse_map)
        else:
            self.map = tf.sparse.from_dense(sparse_map)

    def call(self, inputs):
        if self.mode == "dense":
            return inputs @ self.map
        else:
            input_dims = inputs.shape
            input_matrix = tf.reshape(inputs, (-1, input_dims[-1]))
            out = tf.sparse.sparse_dense_matmul(input_matrix, self.map)
            return tf.reshape(out, input_dims[:-1].as_list() + [-1])


class TFAdaptiveAvgPool2D(tf.keras.layers.Layer):
    def __init__(self, output_shape, mode="dense", **kwargs):
        super().__init__(**kwargs)
        self.w_pool = TFAdaptiveAvgPool1D(output_shape[1], mode=mode)

    def call(self, inputs):
        # Rearrange from NHWC -> NCHW
        inputs = tf.transpose(inputs, perm=[0, 3, 1, 2])
        # Perform W-pooling
        inputs = self.w_pool(inputs)


pt_2d_pooler = torch.nn.AdaptiveAvgPool2d((1, 1))
tf_1d_pooler = TFAdaptiveAvgPool1D(output_dim=1, mode="dense")

# For image size 224
N, C, H, W = (3, 10, 56, 56)
np_input_224_56 = np.random.random(size=(N, C, H, W))
pt_input_224_56 = torch.tensor(np_input_224_56)
tf_input_224_56 = tf.constant(np_input_224_56)

# For image size 32
N, C, H, W = (3, 10, 8, 8)
np_input_32_8 = np.random.random(size=(N, C, H, W))
pt_input_32_8 = torch.tensor(np_input_32_8)
tf_input_32_8 = tf.constant(np_input_32_8)

# 1st run: pt OK
pt_o = pt_2d_pooler(pt_input_224_56)
print(pt_o.shape)

# 1st run: tf OK
tf_o = tf_1d_pooler(tf_input_224_56)
print(tf_o.shape)
print(f"tf_1d_pooler.map has shape: {tf_1d_pooler.map.shape}")


# 2nd run: pt OK
pt_o = pt_2d_pooler(pt_input_32_8)
print(pt_o.shape)

# 2nd run: tf failed
tf_o = tf_1d_pooler(tf_input_32_8)
print(tf_o.shape)


```

Thank you so much @ydshieh! Really appreciate this. 

> Regarding the test, the situation is similar: the method dummy_inputs is called in from_pretrained method. While hard-coded with 224, the model can't run with other image size anymore. Even if you add image_size argument in RegNetConfig, once a
TFRegNetModel is run with an input, it can't run with other input shape anymore. This contradicts to the PyTorch implementation.

I see. I am still a little unsure as to why the cross tests in TF would run then. 

Also ccing @Rocketknight1 for https://github.com/huggingface/transformers/pull/17554#issuecomment-1159556670.


> I see. I am still a little unsure as to why the cross tests in TF would run then.

It runs successfully only if the config has the argument `image_size` (..right?). Is this where you have the question?



It runs successfully (the cross-test) with the `image_size` specified in the config. If that is the case, why the PyTorch cross-test would fail. This is my question. Sorry if it wasn't clear previously. 

In `RegNetModelTester`, the `get_config` method doesn't use `image_size`.

https://github.com/huggingface/transformers/blob/6fdcc6dcb5b9396aa4481513c69cc89a22c5533f/tests/models/regnet/test_modeling_regnet.py#L84-L92

Even if the `image_size` argument is added to `RegNetConfig` with a default value `224`, the test doesn't pass `self.image_size (32)` to it.
Therefore, the TF model will get `224` for `dummy_inputs`, but  the subsequent calls in the test prepare image size 32 for testing. That's why it fails.

In TF test, you (also) added `image_size=self.image_size, ` to `get_config`

https://github.com/huggingface/transformers/blob/6fdcc6dcb5b9396aa4481513c69cc89a22c5533f/tests/models/regnet/test_modeling_tf_regnet.py#L81-L90

which is why it works.

Thanks, @ydshieh. That solved the problem. 

@sgugger @Rocketknight1 the tests should pass now but @ydshieh pointed out a potential concern here: https://github.com/huggingface/transformers/pull/17554#issuecomment-1159556670. Does it make sense to tackle it in a separate PR given Adaptive Average Pooling impacts quite a few models (RegNet, ResNet, Swin, etc.)? 

Ugh, yes. The layer precomputes a map for the input and output shapes at build() time, and will break if you pass inputs with different shapes to the layer afterwards.

With the implementation as written, I think it will be quite difficult to generate the `sparse_map` in the `call()` method. The reason is that there's a lot of computation that's just happening on the CPU to make it, which is fine as a once-off task in the `init`, but that won't really work if it  compiled into the graph.

I think this is a sign that we might have to write a proper `AdaptivePool` op for TF and make a PR to TFA, which I was talking with @amyeroberts about.

> With the implementation as written, I think it will be quite difficult to generate the `sparse_map` in the `call()` method.

Do you think it is necessary to keep the logic for the sparse part?

We can absolutely drop the actual sparse matrix, however we'll still need to compute the dense matrix (which is called `sparse_map` because it's mostly zeros)

@Rocketknight1 WDYT we should do about this PR given the current state? Should we wait for ...

> I think this is a sign that we might have to write a proper AdaptivePool op for TF and make a PR to TFA, which I was talking with @amyeroberts about.

... or is there anything I can do at my end? 

@sayakpaul Hang on! I have a new implementation for `AdaptivePool` that I think will resolve some of these issues, and I should be able to finish it by today.

@sayakpaul I have an implementation [here](https://gist.github.com/Rocketknight1/b0baa8236f379b811fc6bce3da05cc2b), working on testing and optimizations now

@sayakpaul added comments as requested: https://gist.github.com/Rocketknight1/efc47242914788def0144b341b1ad638

> @sayakpaul added comments as requested: https://gist.github.com/Rocketknight1/efc47242914788def0144b341b1ad638

Just read through. Excellent 👌

@Rocketknight1 I tested your new implementation locally ([branch](https://github.com/ariG23498/transformers/tree/new-adaptive-pooler)). 

It works across both TF and PT cross-tests. It also works without `image_size` being supplied to `RegNetConfig()` in the PT test (see [here](https://github.com/ariG23498/transformers/blob/new-adaptive-pooler/tests/models/regnet/test_modeling_regnet.py#L86)).

Cc: @ydshieh 

@sgugger @Rocketknight1 here are the updates:

* Removed `image_size` argument everywhere relevant to avoid any confusion. 
* Used @Rocketknight1's new implementation of adaptive average pooling (https://github.com/huggingface/transformers/pull/17554#issuecomment-1160654493).

Tests are all passing. Let me know if there is anything left on my end.

The TF weights need to be uploaded to the model repository on Hub. After that, I will remove the `from_pt` argument [here](https://github.com/ariG23498/transformers/blob/aritra-regnets/tests/models/regnet/test_modeling_tf_regnet.py#L251) and [here](https://github.com/ariG23498/transformers/blob/aritra-regnets/tests/models/regnet/test_modeling_tf_regnet.py#L275).  

Please also incorporate the updates made in #17731 

> Please also incorporate the updates made in #17731

Copy-paste the changes and add you as a co-author of the commits?

No I think @NielsRogge means making sure you apply on TFRegNet the changes he is working on for all other TF models.

> Please also incorporate the updates made in https://github.com/huggingface/transformers/pull/17731

@NielsRogge could you be a bit more specific? 

RegNets don't have `to_2tuple()`. I understand what needs to be changed regarding `num_channels` but it's hard to isolate that from your PR. If you could point me to the relevant places in this PR that need changing (with respect to yours), that'd be helpful. 

@Rocketknight1 @amyeroberts I have updated the use of Adaptive Average Pooling locally but when I tried running the tests (`RUN_SLOW=1 RUN_PT_TF_CROSS_TESTS=1 python -m pytest tests/models/regnet/test_modeling_tf_regnet.py`) I'm running into:

```
/Users/sayakpaul/.local/bin/.virtualenvs/hf/lib/python3.8/site-packages/pytest_sugar.py:169: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
  xdist_version = LooseVersion(xdist.__version__)
/Users/sayakpaul/.local/bin/.virtualenvs/hf/lib/python3.8/site-packages/pytest_sugar.py:170: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
  if xdist_version >= LooseVersion('1.14'):
Test session starts (platform: darwin, Python 3.8.2, pytest 7.0.0, pytest-sugar 0.9.4)
rootdir: /Users/sayakpaul/Downloads/transformers, configfile: setup.cfg
plugins: dash-2.1.0, sugar-0.9.4, picked-0.4.6, xdist-2.5.0, forked-1.4.0, timeout-2.1.0, hypothesis-6.36.1
collecting ... 
――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――― ERROR collecting tests/models/regnet/test_modeling_tf_regnet.py ――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――
ImportError while importing test module '/Users/sayakpaul/Downloads/transformers/tests/models/regnet/test_modeling_tf_regnet.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/importlib/__init__.py:127: in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
tests/models/regnet/test_modeling_tf_regnet.py:25: in <module>
    from ...test_configuration_common import ConfigTester
tests/test_configuration_common.py:26: in <module>
    from huggingface_hub import HfFolder, Repository, delete_repo, set_access_token
E   ImportError: cannot import name 'set_access_token' from 'huggingface_hub' (/Users/sayakpaul/.local/bin/.virtualenvs/hf/lib/python3.8/site-packages/huggingface_hub/__init__.py)
------------------------------------------------------------------------------------------------------- Captured stderr --------------------------------------------------------------------------------------------------------
WARNING:absl:No GPU/TPU found, falling back to CPU. (Set TF_CPP_MIN_LOG_LEVEL=0 and rerun for more info.)

======================================================================================================= warnings summary =======================================================================================================
../../.local/bin/.virtualenvs/hf/lib/python3.8/site-packages/flatbuffers/compat.py:19
  /Users/sayakpaul/.local/bin/.virtualenvs/hf/lib/python3.8/site-packages/flatbuffers/compat.py:19: DeprecationWarning: the imp module is deprecated in favour of importlib; see the module's documentation for alternative uses
    import imp

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
=================================================================================================== short test summary info ====================================================================================================
FAILED tests/models/regnet/test_modeling_tf_regnet.py
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! Interrupted: 1 error during collection !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

Results (0.77s):
```

@amyeroberts since you incorporated similar changed in https://github.com/huggingface/transformers/pull/17427 yesterday, wondering if you faced something similar. If so, what was the solution? 

> RegNets don't have to_2tuple(). I understand what needs to be changed regarding num_channels but it's hard to isolate that from your PR. If you could point me to the relevant places in this PR that need changing (with respect to yours), that'd be helpful.

So we'd like to add a sanity check regarding `num_channels` to all vision models for which it is relevant.

In PyTorch, I add the following to the models that have an initial embedding layer (stem):
```
num_channels = pixel_values.shape[1]
if num_channels != self.num_channels:
    raise ValueError(
        "Make sure that the channel dimension of the pixel values match with the one set in the configuration."
    )
```
However, when adding something similar to TensorFlow models, like `TFViTMAE`, this failed when using `save_pretrained`. @ydshieh pointed me that it is due to graph mode. So he suggests to use the following for TF models:
```
if tf.executing_eagerly() and num_channels != self.num_channels:
    raise ValueError(
        "Make sure that the channel dimension of the pixel values match with the one set in the configuration."
    )
```

Thanks for explaining. It's clear now.

```if tf.executing_eagerly() and num_channels != self.num_channels:
    raise ValueError(
        "Make sure that the channel dimension of the pixel values match with the one set in the configuration."
    )
```

Where did you add it and how did you obtain `num_channels`? Similarly as `num_channels = pixel_values.shape[1]`? @ydshieh would `shape_list()` have been a better choice for TF models here? 

Well, I could not find the occurrence of `num_channels = pixel_values.shape[1]` in my local clone. My main suggestion (@amyeroberts  mentioned) is about adding `if tf.executing_eagerly()` in the condition.

Also with a discussion with Niels, we found `if getattr(height, "numpy", None) and getattr(width, "numpy", None):` should be replace by `if tf.executing_eagerly()` -- that was my bad when I added TF ViT model.

If the check is being added from the `modeling_*.py` scripts, then I think it's better to access the number of channels with `self.config.num_channels`. 

@Rocketknight1 @amyeroberts regarding https://github.com/huggingface/transformers/pull/17554#issuecomment-1164086078, I was able to fix it by upgrading `huggingface_hub`: `pip install -U huggingface_hub`. 

I have incorporated the rest of the changes:

* Adaptive average pool -> Global average pool
* Changes as per https://github.com/huggingface/transformers/pull/17554#issuecomment-1164094973

Cc: @NielsRogge @sgugger 

This part is pending:

> The TF weights need to be uploaded to the model repository on Hub. After that, I will remove the from_pt argument [here](https://github.com/ariG23498/transformers/blob/aritra-regnets/tests/models/regnet/test_modeling_tf_regnet.py#L251) and [here](https://github.com/ariG23498/transformers/blob/aritra-regnets/tests/models/regnet/test_modeling_tf_regnet.py#L275).


@sgugger @Rocketknight1 anything pending on my end to move this ahead?

Thank you! 

> The TF weights need to be uploaded to the model repository on Hub. After that, I will remove the from_pt argument here and here.

This part is pending then. 

As soon as @ariG23498 performs a rebase the errors should go away (I can't since the forked repo's owner can only do that I guess).

Hey folks! The hub PRs for most models are open (the exception is the 10b models, which I'm having a look at), you should be able to remove the `from_pt` soon :)

Thanks, @gante! Keep us posted. 



@gante thanks for your hard work on getting the TF parameters of this model to Hub (there are a total of 32 checkpoints in case anyone's curious). I really appreciate this help!

I have removed the occurrences of `from_pt=True` from TF test script. With this, I think the PR is ready to merge an exception on the 10B checkpoint of the model. It requires some functionalities that are apparently missing but will be likely added soon. 

The failing tests seem to be unrelated to this PR?

Yes, pytorch v1.12 was released a few hours ago and broke a few things here. [We have pinned pytorch to <1.12](https://github.com/huggingface/transformers/blob/main/setup.py#L163) -- rebasing with `main` should fix the problems :)

@gante we performed a rebase but the `pipelines` test for Torch seems to be still failing. 

That error seems unrelated to this PR, so I think you could probably merge anyway.

Over to you @gante then since I can't merge :D 

The error is on `main` as well -- merging 🤞