Fix the issue of blip model returning loss even when the label is not provided. 
Author: raghavanone
Pull request: yes

# What does this PR do?

Fixes #21510 

@NielsRogge 

Comments:
_The documentation is not available anymore as the PR was closed or merged._

cc @younesbelkada and @amyeroberts 

> 

@younesbelkada 
One of the test (test_inference_image_captioning_fp16) is failing, not sure if it is related to my changes . 


Hi @raghavanone 
Hum I just went through the daily CI report and this test seems to be not failing on our end, can you share with us the traceback of the error?

> Hi @raghavanone Hum I just went through the daily CI report and this test seems to be not failing on our end, can you share with us the traceback of the error?

```
tests/models/blip/test_modeling_blip.py:1113 (BlipModelIntegrationTest.test_inference_image_captioning_fp16)
self = <tests.models.blip.test_modeling_blip.BlipModelIntegrationTest testMethod=test_inference_image_captioning_fp16>

    def test_inference_image_captioning_fp16(self):
        model = BlipForConditionalGeneration.from_pretrained(
            "Salesforce/blip-image-captioning-base", torch_dtype=torch.float16
        ).to(torch_device)
        processor = BlipProcessor.from_pretrained("Salesforce/blip-image-captioning-base")
        image = prepare_img()
    
        # image only
        inputs = processor(images=image, return_tensors="pt").to(torch_device, torch.float16)
    
>       predictions = model.generate(**inputs)

tests/models/blip/test_modeling_blip.py:1124: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
/opt/homebrew/Caskroom/miniforge/base/envs/hf_dev/lib/python3.8/site-packages/torch/autograd/grad_mode.py:27: in decorate_context
    return func(*args, **kwargs)
src/transformers/models/blip/modeling_blip.py:1068: in generate
    vision_outputs = self.vision_model(
/opt/homebrew/Caskroom/miniforge/base/envs/hf_dev/lib/python3.8/site-packages/torch/nn/modules/module.py:1194: in _call_impl
    return forward_call(*input, **kwargs)
src/transformers/models/blip/modeling_blip.py:694: in forward
    hidden_states = self.embeddings(pixel_values)
/opt/homebrew/Caskroom/miniforge/base/envs/hf_dev/lib/python3.8/site-packages/torch/nn/modules/module.py:1194: in _call_impl
    return forward_call(*input, **kwargs)
src/transformers/models/blip/modeling_blip.py:241: in forward
    patch_embeds = self.patch_embedding(pixel_values)  # shape = [*, width, grid, grid]
/opt/homebrew/Caskroom/miniforge/base/envs/hf_dev/lib/python3.8/site-packages/torch/nn/modules/module.py:1194: in _call_impl
    return forward_call(*input, **kwargs)
/opt/homebrew/Caskroom/miniforge/base/envs/hf_dev/lib/python3.8/site-packages/torch/nn/modules/conv.py:463: in forward
    return self._conv_forward(input, self.weight, self.bias)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = Conv2d(3, 768, kernel_size=(16, 16), stride=(16, 16))
input = tensor([[[[ 0.8647,  0.9229,  0.9375,  ...,  1.7549,  1.7549,  1.7549],
          [ 0.9082,  0.9375,  0.9521,  ...,  1....2856, -0.3569],
          [-0.3142, -0.3425, -0.3569,  ..., -0.3000, -0.3569, -0.3994]]]],
       dtype=torch.float16)
weight = Parameter containing:
tensor([[[[ 3.3875e-03,  1.4102e-04,  7.0906e-04,  ..., -4.6539e-03,
            1.2560e-03, -5....3.7727e-03,  ..., -1.3084e-03,
            4.8304e-04,  7.3357e-03]]]], dtype=torch.float16,
       requires_grad=True)
bias = Parameter containing:
tensor([ 7.6477e-02,  7.6233e-02,  2.6343e-01,  2.6718e-02,  3.8727e-02,
         1.2962e-02, -2....6932e-02, -9.2529e-02,
         7.5012e-02,  6.4812e-03, -1.7303e-02], dtype=torch.float16,
       requires_grad=True)

    def _conv_forward(self, input: Tensor, weight: Tensor, bias: Optional[Tensor]):
        if self.padding_mode != 'zeros':
            return F.conv2d(F.pad(input, self._reversed_padding_repeated_twice, mode=self.padding_mode),
                            weight, bias, self.stride,
                            _pair(0), self.dilation, self.groups)
>       return F.conv2d(input, weight, bias, self.stride,
                        self.padding, self.dilation, self.groups)
E       RuntimeError: "slow_conv2d_cpu" not implemented for 'Half'

/opt/homebrew/Caskroom/miniforge/base/envs/hf_dev/lib/python3.8/site-packages/torch/nn/modules/conv.py:459: RuntimeError
```

I see, you are not running the tests on GPU, if you don't have access to any GPU I can run the slow test for you
(Also we might need to add `require_torch_gpu` decorator on this test, if you could also add it in this PR it would be great 🙏 )

> test_inference_image_captioning_fp16

Oh, I just realised that, I am adding the tag. 

@younesbelkada  Did that test fail in your setup ? Is there something I have to fix ?