[CLIP] minor fixes
Author: patil-suraj
Pull request: yes

# What does this PR do?

Few minor fixes in CLIP. Thanks for spotting these @ydshieh !
Fixes #14024




Comments:
Hi, @patil-suraj 

It seems adding `@dataclass` to `CLIPOutput` causes `FlaxCLIPModelTest.test_equivalence_flax_to_pt/test_equivalence_pt_to_flax` failed.

```
>               self.assertEqual(len(fx_outputs), len(pt_outputs), "Output lengths differ between Flax and PyTorch")
E               AssertionError: 6 != 5 : Output lengths differ between Flax and PyTorch
```

I didn't look for the cause though.


Full error outputs
```
self = <tests.test_modeling_flax_clip.FlaxCLIPModelTest testMethod=test_equivalence_flax_to_pt>

    def test_equivalence_flax_to_pt(self):
        config, inputs_dict = self.model_tester.prepare_config_and_inputs_for_common()

        for model_class in self.all_model_classes:
            with self.subTest(model_class.__name__):
                # prepare inputs
                prepared_inputs_dict = self._prepare_for_class(inputs_dict, model_class)
                pt_inputs = {k: torch.tensor(v.tolist()) for k, v in prepared_inputs_dict.items()}

                # load corresponding PyTorch class
                pt_model_class_name = model_class.__name__[4:]  # Skip the "Flax" at the beginning
                pt_model_class = getattr(transformers, pt_model_class_name)

                pt_model = pt_model_class(config).eval()
                fx_model = model_class(config, dtype=jnp.float32)

                pt_model = load_flax_weights_in_pytorch_model(pt_model, fx_model.params)

                # make sure weights are tied in PyTorch
                pt_model.tie_weights()

                with torch.no_grad():
                    pt_outputs = pt_model(**pt_inputs).to_tuple()
                # PyTorch CLIPModel returns loss, we skip it here as we don't return loss in JAX/Flax models
                pt_outputs = pt_outputs[1:]

                fx_outputs = fx_model(**prepared_inputs_dict).to_tuple()
>               self.assertEqual(len(fx_outputs), len(pt_outputs), "Output lengths differ between Flax and PyTorch")
E               AssertionError: 6 != 5 : Output lengths differ between Flax and PyTorch



self = <tests.test_modeling_flax_clip.FlaxCLIPModelTest testMethod=test_equivalence_pt_to_flax>

    def test_equivalence_pt_to_flax(self):
        config, inputs_dict = self.model_tester.prepare_config_and_inputs_for_common()

        for model_class in self.all_model_classes:
            with self.subTest(model_class.__name__):
                # prepare inputs
                prepared_inputs_dict = self._prepare_for_class(inputs_dict, model_class)
                pt_inputs = {k: torch.tensor(v.tolist()) for k, v in prepared_inputs_dict.items()}

                # load corresponding PyTorch class
                pt_model_class_name = model_class.__name__[4:]  # Skip the "Flax" at the beginning
                pt_model_class = getattr(transformers, pt_model_class_name)

                pt_model = pt_model_class(config).eval()
                fx_model = model_class(config, dtype=jnp.float32)

                fx_state = convert_pytorch_state_dict_to_flax(pt_model.state_dict(), fx_model)
                fx_model.params = fx_state

                with torch.no_grad():
                    pt_outputs = pt_model(**pt_inputs).to_tuple()
                # PyTorch CLIPModel returns loss, we skip it here as we don't return loss in JAX/Flax models
                pt_outputs = pt_outputs[1:]

                fx_outputs = fx_model(**prepared_inputs_dict).to_tuple()
>               self.assertEqual(len(fx_outputs), len(pt_outputs), "Output lengths differ between Flax and PyTorch")
E               AssertionError: 6 != 5 : Output lengths differ between Flax and PyTorch


```

I think this PR broke master: https://app.circleci.com/pipelines/github/huggingface/transformers/29119/workflows/de3948c8-e962-4b6e-ad5e-6b0ac8093a1c/jobs/290791 

@sgugger - I'll open a PR to add PT <=> TF, PT<=> Flax, tests tomorrow