Skip `TFCvtModelTest::test_keras_fit_mixed_precision` for now
Author: ydshieh
Pull request: yes

# What does this PR do?

#23339 uses TF 2.12 with CUDA 11.8 (and CUDNN 8700). With those, the test
```bash
tests/models/cvt/test_modeling_tf_cvt.py::TFCvtModelTest::test_keras_fit_mixed_precision
```
gets
```bash
 tensorflow.python.framework.errors_impl.UnknownError: Failed to determine best cudnn convolution algorithm for:
```
and affect other two tests
```bash
FAILED tests/models/cvt/test_modeling_tf_cvt.py::TFCvtModelTest::test_pt_tf_model_equivalence - AssertionError: 0.007014513 not less than or equal to 1e-05 : outputs.last_hidden_state: Difference between torch and tf is 0.00701451301574707 (>= 1e-05).
FAILED tests/models/cvt/test_modeling_tf_cvt.py::TFCvtModelIntegrationTest::test_inference_image_classification_head - AssertionError: False is not true
```
**Those 2 tests will pass if `test_keras_fit_mixed_precision` is not run in the same pytest process.** (probably the GPU/CUDA/CUDNN is in bad states).

We will have to take a look and fix `test_keras_fit_mixed_precision`. But in the meantime, **let's skip it and not to affect the other 2 tests.**

@Rocketknight1 If you ever want to take a look this CUDA/CUDNN/TF issue. (Maybe better to open an issue in TF repo. but it may take 10 years to get a fix)

Comments:
Here is the full error when running
```bash
python3 -m pytest -v tests/models/cvt/test_modeling_tf_cvt.py::TFCvtModelTest::test_keras_fit_mixed_precision
```

## Error
```bash
self = <tests.models.cvt.test_modeling_tf_cvt.TFCvtModelTest testMethod=test_keras_fit_mixed_precision>

    def test_keras_fit_mixed_precision(self):
        policy = tf.keras.mixed_precision.Policy("mixed_float16")
        tf.keras.mixed_precision.set_global_policy(policy)
>       super().test_keras_fit()

tests/models/cvt/test_modeling_tf_cvt.py:192: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
tests/test_modeling_tf_common.py:1585: in test_keras_fit
    history1 = model.fit(
/usr/local/lib/python3.8/dist-packages/keras/utils/traceback_utils.py:70: in error_handler
    raise e.with_traceback(filtered_tb) from None
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <transformers.models.cvt.modeling_tf_cvt.TFCvtForImageClassification object at 0x7f4c0b7f10a0>
data = {'labels': <tf.Tensor: shape=(13,), dtype=int32, numpy=array([0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], dtype=int32)>, '...,
         [0.38049886, 0.97876924, 0.96599656, ..., 0.5474588 ,
          0.8447144 , 0.1995452 ]]]], dtype=float32)>}

    def train_step(self, data):
        """
        A modification of Keras's default `train_step` that correctly handles matching outputs to labels for our models
        and supports directly training on the loss output head. In addition, it ensures input keys are copied to the
        labels where appropriate. It will also copy label keys into the input dict when using the dummy loss, to ensure
        that they are available to the model during the forward pass.
        """
    
        # We hardcode the most common renamings; models with weirder names can set `self._label_to_output_map`
        arg_names = list(dict(inspect.signature(self.call).parameters).keys())
        label_kwargs = find_labels(self.__class__)
        label_to_output = self.get_label_to_output_name_mapping()
        output_to_label = {val: key for key, val in label_to_output.items()}
        if not self._using_dummy_loss and parse(tf.__version__) < parse("2.11.0"):
            # Newer TF train steps leave this out
            data = data_adapter.expand_1d(data)
        x, y, sample_weight = data_adapter.unpack_x_y_sample_weight(data)
        # If the inputs are mutable dictionaries, make a shallow copy of them because we will modify
        # them during input/label pre-processing. This avoids surprising the user by wrecking their data.
        # In addition, modifying mutable Python inputs makes XLA compilation impossible.
        if isinstance(x, dict):
            x = x.copy()
        if isinstance(y, dict):
            y = y.copy()
    
        # When using a dummy loss, we ensure that separate labels are copied to the correct model arguments,
        # if those keys are not already present in the input dict
        if self._using_dummy_loss and y is not None:
            # If y is a tensor and the model only has one label-like input, map y to that input
            if len(label_kwargs) == 1 and isinstance(y, tf.Tensor):
                if isinstance(x, tf.Tensor):
                    x = {arg_names[0]: x}
                label_kwarg = next(iter(label_kwargs))
                if label_kwarg not in x:
                    x[label_kwarg] = y
            # Otherwise, copy keys from y to x as long as they weren't already present in x
            elif isinstance(y, dict):
                if isinstance(x, tf.Tensor):
                    x = {arg_names[0]: x}
                for key, val in y.items():
                    if key in arg_names and key not in x:
                        x[key] = val
                    elif output_to_label.get(key, None) in arg_names and key not in x:
                        x[output_to_label[key]] = val
        if y is None:
            y = {key: val for key, val in x.items() if key in label_kwargs}
            if not y and not self._using_dummy_loss:
                raise ValueError("Could not find label column(s) in input dict and no separate labels were provided!")
    
        if isinstance(y, dict):
            # Rename labels at this point to match output heads
            y = {label_to_output.get(key, key): val for key, val in y.items()}
    
        # Run forward pass.
        with tf.GradientTape() as tape:
            if self._using_dummy_loss and "return_loss" in arg_names:
                y_pred = self(x, training=True, return_loss=True)
            else:
                y_pred = self(x, training=True)
            if self._using_dummy_loss:
                loss = self.compiled_loss(y_pred.loss, y_pred.loss, sample_weight, regularization_losses=self.losses)
            else:
                loss = None
    
            # This next block matches outputs to label keys. Tensorflow's standard method for doing this
            # can get very confused if any of the keys contain nested values (e.g. lists/tuples of Tensors)
            if isinstance(y, dict) and len(y) == 1:
                if list(y.keys())[0] in y_pred.keys():
                    y_pred = y_pred[list(y.keys())[0]]
                elif list(y_pred.keys())[0] == "loss":
                    y_pred = y_pred[1]
                else:
                    y_pred = y_pred[0]
                _, y = y.popitem()
            elif isinstance(y, dict):
                # If the labels are a dict, match keys from the output by name
                y_pred = {key: val for key, val in y_pred.items() if key in y}
            elif isinstance(y, tuple) or isinstance(y, list):
                # If the labels are a tuple/list, match keys to the output by order, skipping the loss.
                if list(y_pred.keys())[0] == "loss":
                    y_pred = y_pred.to_tuple()[1:]
                else:
                    y_pred = y_pred.to_tuple()
                y_pred = y_pred[: len(y)]  # Remove unused fields in case those cause problems
            else:
                # If the labels are a single tensor, match them to the first non-loss tensor in the output
                if list(y_pred.keys())[0] == "loss":
                    y_pred = y_pred[1]
                else:
                    y_pred = y_pred[0]
    
            if loss is None:
                loss = self.compiled_loss(y, y_pred, sample_weight, regularization_losses=self.losses)
    
        # Run backwards pass.
>       self.optimizer.minimize(loss, self.trainable_variables, tape=tape)
E       tensorflow.python.framework.errors_impl.UnknownError: Failed to determine best cudnn convolution algorithm for:
E       %cudnn-conv.6 = (f16[1,3,3,96]{3,2,1,0}, u8[0]{0}) custom-call(f16[1,5,5,1248]{3,2,1,0} %bitcast.46, f16[96,3,3,16]{3,2,1,0} %transpose.3), window={size=3x3}, dim_labels=b01f_o01i->b01f, feature_group_count=96, custom_call_target="__cudnn$convForward", metadata={op_type="Conv2DBackpropFilter" op_name="gradients/Conv2D_grad/Conv2DBackpropFilter" source_file="/usr/local/lib/python3.8/dist-packages/keras/layers/convolutional/base_conv.py" source_line=286}, backend_config="{\"conv_result_scale\":1,\"activation_mode\":\"0\",\"side_input_scale\":0}"
E       
E       Original error: UNKNOWN: CUDNN_STATUS_BAD_PARAM
E       in tensorflow/compiler/xla/stream_executor/cuda/cuda_dnn.cc(3588): 'op' CUDNN_BACKEND_OPERATION: cudnnFinalize Failed
E       
E       To ignore this failure and try to use a fallback algorithm (which may have suboptimal performance), use XLA_FLAGS=--xla_gpu_strict_conv_algorithm_picker=false.  Please also file a bug for the root cause of failing autotuning. [Op:__inference___backward__jit_compiled_convolution_op_11189_11200]

src/transformers/modeling_tf_utils.py:1611: UnknownError
```

_The documentation is not available anymore as the PR was closed or merged._

Ugh, this seems like a nightmare upstream issue alright - I think it's the best use of our time to just leave it unless it starts affecting multiple models, or if we start seeing it outside of the mixed_precision code paths.

Actually, it is mixed precision with training. I didn't see any other TF model having this test. Good for me to ignore it!