Load `instructblip-flan-t5-xl` changes the results of `instructblip-vicuna-7b`
Author: ydshieh
Pull request: no

When loading `instructblip-flan-t5-xl` using `InstructBlipForConditionalGeneration`, it will change the (logits) output of `instructblip-vicuna-7b`. See the reproduction section.

If I tried to load `facebook/opt-350m` (using `OPTForCausalLM`), it won't cause this situation (i.e. `instructblip-vicuna-7b` have the same logits).

If I load `instructblip-flan-t5-xl`, it won't change the results of `facebook/opt-350m` (loaded in 8-bit).

Something is strange here and requires further investigation.

### Reproduction

The following one shows `Salesforce/instructblip-vicuna-7b` is affected by `instructblip-flan-t5-xl`

```
import requests
import torch
from PIL import Image
from transformers import InstructBlipProcessor, InstructBlipForConditionalGeneration, OPTForCausalLM

torch_device = "cuda"

url = "https://raw.githubusercontent.com/salesforce/LAVIS/main/docs/_static/Confusing-Pictures.jpg"
image = Image.open(requests.get(url, stream=True).raw).convert("RGB")
prompt = "What is unusual about this image?"

processor = InstructBlipProcessor.from_pretrained("Salesforce/instructblip-vicuna-7b")
inputs = processor(images=image, text=prompt, return_tensors="pt").to(torch_device, torch.float16)

def check_vicuna_logits(load_flan_t5=False, load_opt=False):

    if load_flan_t5:
        _ = InstructBlipForConditionalGeneration.from_pretrained(
           "Salesforce/instructblip-flan-t5-xl",
           # don't matter `float16` or `bfloat16`
           torch_dtype=torch.bfloat16,
        )

    if load_opt:
        _ = OPTForCausalLM.from_pretrained(
           "facebook/opt-350m",
           # don't matter `float16` or `bfloat16`
           torch_dtype=torch.bfloat16,
        )

    model = InstructBlipForConditionalGeneration.from_pretrained(
        "Salesforce/instructblip-vicuna-7b", load_in_8bit=True
    )
    with torch.no_grad():
        logits = model(**inputs).logits

    print(logits[0, :3, :3])


# [ -3.4902, -12.5078,   8.4141]
check_vicuna_logits(load_flan_t5=False)

# [ -3.4902, -12.5078,   8.4141]
check_vicuna_logits(load_flan_t5=False, load_opt=True)

# Once `Salesforce/instructblip-flan-t5-xl` is loaded (even just once), `logits` changes to `-3.4609`.
# [ -3.4609, -12.0156,   8.3750]
check_vicuna_logits(load_flan_t5=True)

# continue to be `-3.4609`
# [ -3.4609, -12.0156,   8.3750]
check_vicuna_logits(load_flan_t5=False)
```

The following one shows `Opt` won't be affected by `/instructblip-flan-t5-xl`

```python
import torch

from transformers import InstructBlipForConditionalGeneration, OPTForCausalLM, AutoTokenizer

torch_device = "cuda"
tokenizer = AutoTokenizer.from_pretrained("facebook/opt-350m")

prompt = "Hey, are you conscious? Can you talk to me?"
inputs = tokenizer(prompt, return_tensors="pt")   # .to(torch_device, torch.float16)

for k, v in inputs.items():
    if torch.is_floating_point(v):
        inputs[k] = v.to(device=torch_device, dtype=torch.bfloat16)

def check_opt_logits(load_flan_t5=False):

    if load_flan_t5:
        _ = InstructBlipForConditionalGeneration.from_pretrained(
           "Salesforce/instructblip-flan-t5-xl",
           # don't matter `float16` or `bfloat16`
           torch_dtype=torch.bfloat16,
        )

    model = OPTForCausalLM.from_pretrained(
       "facebook/opt-350m",
       load_in_8bit=True,
    )

    with torch.no_grad():
        logits = model(**inputs).logits

    print(logits[0, :3, :3])

check_opt_logits(load_flan_t5=False)
check_opt_logits(load_flan_t5=True)
```


### Expected behavior

The outputs of `Salesforce/instructblip-vicuna-7b` **SHOULD NOT** be affected by if we load `instructblip-flan-t5-xl` or not.

Comments: