Fix pipeline tests - don't set torch_dtype on non-torch pipelines
Author: amyeroberts
Pull request: yes

# What does this PR do?

Fixes errors introduced by the merge of #31342 

Example failing run: https://app.circleci.com/pipelines/github/huggingface/transformers/97170/workflows/fd23720b-5884-42d5-8877-e69afcdeda34/jobs/1285703

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_31809). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Hi @amyeroberts Thanks for this.

- (more of a nit, see next point too) It makes a lot of sense not default to `float32`. However, we have many `def get_test_pipeline(self, model, tokenizer, processor, torch_dtype="float32")`, and I believe it will make even more sense if we change all those default values to `None`.

- (**IMPORTANT**) In the tests, we are doing/using

```
    def get_test_pipeline(self, model, tokenizer, processor, torch_dtype="float32"):
        classifier = ZeroShotClassificationPipeline(
            model=model, tokenizer=tokenizer, candidate_labels=["polics", "health"], torch_dtype=torch_dtype
        )
```
But if we look the definition of `class Pipeline`, `torch_dtype` is never used. It's actually defined a property with 

```python
    @property
    def torch_dtype(self) -> Optional["torch.dtype"]:
        """
        Torch dtype of the model (if it's Pytorch model), `None` otherwise.
        """
        return getattr(self.model, "dtype", None)
```

So we are not actually testing properly with fp32/fp16 as we expect. (I have to verify to make sure). This is important to address, either in this PR or another.

- And the failing is when `pipeline(...., torch_dtype)` is used. I fixed it in #31827. 




I confirm the 2nd point in the above comment: we are not testing the torch_dtype properly, by

- running : `python3 -m pytest -v tests/models/blip/test_modeling_blip.py::BlipModelTest::test_pipeline_image_to_text_fp16`
- breakpoint around: `tests/pipelines/test_pipelines_image_to_text.py` line 54, after `pipe` defined.

It shows

```bash
(Pdb) pipe.model.dtype
torch.float32
```

More details
```
root@21530a8f5c8d:/transformers# python3 -m pytest -v tests/models/blip/test_modeling_blip.py::BlipModelTest::test_pipeline_image_to_text_fp16

============================================================================= test session starts ==============================================================================
platform linux -- Python 3.8.10, pytest-7.4.4, pluggy-1.5.0 -- /usr/bin/python3
cachedir: .pytest_cache
hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase(PosixPath('/transformers/.hypothesis/examples'))
rootdir: /transformers
configfile: pyproject.toml
plugins: typeguard-4.2.1, hydra-core-1.3.2, xdist-3.6.1, rich-0.1.1, dash-2.17.0, timeout-2.3.1, hypothesis-6.102.6
collected 1 item                                                                                                                                                               

tests/models/blip/test_modeling_blip.py::BlipModelTest::test_pipeline_image_to_text_fp16 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> PDB set_trace (IO-capturing turned off) >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> /transformers/tests/pipelines/test_pipelines_image_to_text.py(54)get_test_pipeline()
-> Image.open("./tests/fixtures/tests_samples/COCO/000000039769.png"),
(Pdb) pipe
<transformers.pipelines.image_to_text.ImageToTextPipeline object at 0x7f0efc188fa0>
(Pdb) pipe.model.dtype
torch.float32
(Pdb) 
```