Error: C:\Users\Dell\AI\pythonProject\.venv\Scripts\python.exe C:\Users\Dell\AI\pythonProject\!EridAI\VLM\main.py 
2024-10-06 17:36:09.357446: I tensorflow/core/util/port.cc:153] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
2024-10-06 17:36:11.614208: I tensorflow/core/util/port.cc:153] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
Map (num_proc=4): 100%|██████████| 2434/2434 [00:35<00:00, 68.23 examples/s] 
Converted image at index 0 to tensor.
Converted image at index 0 to tensor.
Converted image at index 1 to tensor.
Converted image at index 0 to tensor.
Converted image at index 1 to tensor.
Converted image at index 2 to tensor.
Converted image at index 1 to tensor.
Converted image at index 0 to tensor.
Converted image at index 2 to tensor.
Converted image at index 2 to tensor.
Converted image at index 1 to tensor.
Converted image at index 3 to tensor.
Converted image at index 3 to tensor.
Converted image at index 3 to tensor.
Converted image at index 2 to tensor.
Converted image at index 4 to tensor.
Converted image at index 4 to tensor.
Converted image at index 4 to tensor.
Converted image at index 3 to tensor.
Converted image at index 5 to tensor.
Converted image at index 5 to tensor.
Converted image at index 5 to tensor.
Converted image at index 4 to tensor.
Converted image at index 6 to tensor.
Converted image at index 6 to tensor.
Converted image at index 6 to tensor.
Converted image at index 5 to tensor.
Converted image at index 7 to tensor.
Converted image at index 7 to tensor.
Converted image at index 7 to tensor.
Converted image at index 6 to tensor.
Converted image at index 7 to tensor.
Converted image at index 0 to tensor.
Converted image at index 0 to tensor.
Converted image at index 1 to tensor.
Converted image at index 1 to tensor.
Converted image at index 2 to tensor.
Converted image at index 0 to tensor.
Converted image at index 2 to tensor.
Converted image at index 3 to tensor.
Converted image at index 0 to tensor.
Converted image at index 1 to tensor.
Converted image at index 3 to tensor.
Converted image at index 4 to tensor.
Converted image at index 1 to tensor.
Converted image at index 2 to tensor.
Converted image at index 4 to tensor.
Converted image at index 5 to tensor.
Converted image at index 2 to tensor.
Converted image at index 3 to tensor.
Converted image at index 6 to tensor.
Converted image at index 5 to tensor.
Converted image at index 3 to tensor.
Converted image at index 4 to tensor.
Converted image at index 7 to tensor.
Converted image at index 6 to tensor.
Converted image at index 4 to tensor.
Converted image at index 5 to tensor.
Converted image at index 7 to tensor.
Converted image at index 5 to tensor.
Converted image at index 6 to tensor.
Converted image at index 6 to tensor.
Converted image at index 7 to tensor.
Converted image at index 7 to tensor.
C:\Users\Dell\AI\pythonProject\.venv\Lib\site-packages\transformers\models\clip\modeling_clip.py:480: UserWarning: 1Torch was not compiled with flash attention. (Triggered internally at ..\aten\src\ATen\native\transformers\cuda\sdp_utils.cpp:455.)
  attn_output = torch.nn.functional.scaled_dot_product_attention(
Converted image at index 0 to tensor.
Converted image at index 1 to tensor.
Converted image at index 2 to tensor.
Converted image at index 3 to tensor.
Converted image at index 4 to tensor.
Converted image at index 5 to tensor.
Converted image at index 6 to tensor.
Converted image at index 7 to tensor.
Traceback (most recent call last):
  File "C:\Users\Dell\AI\pythonProject\!EridAI\VLM\main.py", line 220, in <module>
    main()
  File "C:\Users\Dell\AI\pythonProject\!EridAI\VLM\main.py", line 148, in main
    outputs = model(input_ids=input_ids, attention_mask=attention_mask, image=images)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Dell\AI\pythonProject\.venv\Lib\site-packages\torch\nn\modules\module.py", line 1532, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Dell\AI\pythonProject\.venv\Lib\site-packages\torch\nn\modules\module.py", line 1541, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Dell\AI\pythonProject\!EridAI\VLM\main.py", line 23, in forward
    outputs = self.gpt2(
              ^^^^^^^^^^
  File "C:\Users\Dell\AI\pythonProject\.venv\Lib\site-packages\torch\nn\modules\module.py", line 1532, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Dell\AI\pythonProject\.venv\Lib\site-packages\torch\nn\modules\module.py", line 1541, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Dell\AI\pythonProject\.venv\Lib\site-packages\transformers\models\gpt2\modeling_gpt2.py", line 1315, in forward
    transformer_outputs = self.transformer(
                          ^^^^^^^^^^^^^^^^^
  File "C:\Users\Dell\AI\pythonProject\.venv\Lib\site-packages\torch\nn\modules\module.py", line 1532, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Dell\AI\pythonProject\.venv\Lib\site-packages\torch\nn\modules\module.py", line 1541, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Dell\AI\pythonProject\.venv\Lib\site-packages\transformers\models\gpt2\modeling_gpt2.py", line 1157, in forward
    hidden_states = hidden_states.view(output_shape)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
RuntimeError: cannot reshape tensor of 0 elements into shape [-1, 0, 768] because the unspecified dimension size -1 can be any value and is ambiguous

Process finished with exit code 1
 Code: import torch
from torch import nn
from torch.utils.data import DataLoader
from transformers import GPT2Tokenizer, GPT2LMHeadModel, CLIPProcessor, CLIPModel
from datasets import load_dataset
from PIL import Image


class MultimodalGPT2(nn.Module):
    def __init__(self, gpt2_model_name, clip_model_name):
        super(MultimodalGPT2, self).__init__()
        self.clip = CLIPModel.from_pretrained(clip_model_name)
        self.gpt2 = GPT2LMHeadModel.from_pretrained(gpt2_model_name)
        self.adapter = nn.Linear(self.clip.config.projection_dim, self.gpt2.config.n_embd)

    def forward(self, input_ids, attention_mask, image=None):
        if image is not None:
            image_features = self.clip.get_image_features(pixel_values=image)
            adapter_features = self.adapter(image_features)
            adapter_features = adapter_features.unsqueeze(1).expand(-1, input_ids.size(1), -1)
            token_embeddings = self.gpt2.transformer.wte(input_ids)
            inputs_embeds = token_embeddings + adapter_features
            outputs = self.gpt2(
                inputs_embeds=inputs_embeds,
                attention_mask=attention_mask,
                labels=input_ids
            )
        else:
            outputs = self.gpt2(
                input_ids=input_ids,
                attention_mask=attention_mask,
                labels=input_ids
            )
        return outputs


def preprocess_data(example, processor, tokenizer):
    images = example.get('images', [])
    if images:
        try:
            image = images[0]
            if isinstance(image, Image.Image):
                image = image.convert('RGB')
            else:
                image = Image.open(image).convert('RGB')
            # Process image with CLIPProcessor
            encoding = processor(images=image, return_tensors="pt")
            image_tensor = encoding['pixel_values']

            # Ensure pixel_values have shape (1, C, H, W) before squeezing
            if image_tensor.dim() == 4 and image_tensor.size(0) == 1:
                image_tensor = image_tensor.squeeze(0)  # Now shape: (C, H, W)
            else:
                raise ValueError(f"Unexpected pixel_values shape: {image_tensor.shape}")

            # Assertions for debugging
            assert isinstance(image_tensor, torch.Tensor), f"Image is not a tensor: {type(image_tensor)}"
            assert image_tensor.dim() == 3, f"Image tensor has incorrect dimensions: {image_tensor.dim()}"

        except Exception as e:
            print(f"Failed to process image: {e}")
            image_tensor = None
    else:
        image_tensor = None

    # Combine 'user' and 'assistant' texts
    text = (example.get('user', '') + " " + example.get('assistant', '')).strip()
    encoding = tokenizer(
        text, truncation=True, padding='max_length', max_length=128
    )

    return {
        'input_ids': encoding['input_ids'],
        'attention_mask': encoding['attention_mask'],
        'image': image_tensor
    }


def collate_fn(batch):
    # Explicitly specify dtype for input_ids and attention_mask
    input_ids = torch.tensor([item['input_ids'] for item in batch], dtype=torch.long)
    attention_mask = torch.tensor([item['attention_mask'] for item in batch], dtype=torch.long)

    images = []
    for idx, item in enumerate(batch):
        img = item['image']
        if img is not None:
            if isinstance(img, torch.Tensor):
                images.append(img)
            else:
                try:
                    img = torch.tensor(img, dtype=torch.float)  # Ensure float type for images
                    images.append(img)
                    print(f"Converted image at index {idx} to tensor.")
                except Exception as e:
                    print(f"Failed to convert image at index {idx}: {e}")
                    continue

    if images:
        try:
            images = torch.stack(images)
        except Exception as e:
            print(f"Failed to stack images: {e}")
            images = None
    else:
        images = None

    return {'input_ids': input_ids, 'attention_mask': attention_mask, 'image': images}


def main():
    device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
    dataset = load_dataset('HuggingFaceM4/the_cauldron', 'ai2d', split='train')

    tokenizer = GPT2Tokenizer.from_pretrained('gpt2')
    tokenizer.pad_token = tokenizer.eos_token

    processor = CLIPProcessor.from_pretrained('openai/clip-vit-base-patch32')

    # Preprocess the dataset
    processed_dataset = dataset.map(
        lambda x: preprocess_data(x, processor, tokenizer),
        remove_columns=dataset.column_names,
        num_proc=4  # Utilize multiple processes if possible
    )

    # Create DataLoader with enhanced collate_fn
    dataloader = DataLoader(
        processed_dataset,
        batch_size=8,
        shuffle=True,
        collate_fn=collate_fn,
        num_workers=4,  # Adjust based on your CPU cores
        pin_memory=True  # Speeds up data transfer to GPU
    )

    model = MultimodalGPT2('gpt2', 'openai/clip-vit-base-patch32').to(device)

    optimizer = torch.optim.AdamW(model.parameters(), lr=5e-5)  # Use PyTorch's AdamW

    model.train()
    for epoch in range(1):
        for batch_idx, batch in enumerate(dataloader):
            input_ids = batch['input_ids'].to(device)
            attention_mask = batch['attention_mask'].to(device)
            images = batch['image'].to(device) if batch['image'] is not None else None

            outputs = model(input_ids=input_ids, attention_mask=attention_mask, image=images)
            loss = outputs.loss
            loss.backward()
            optimizer.step()
            optimizer.zero_grad()
            print(f"Epoch {epoch + 1}, Batch {batch_idx + 1}: Loss = {loss.item()}")

    torch.save(model.state_dict(), 'multimodal_gpt2.pth')
    model.eval()

    def generate(prompt_text=None, image_path=None):
        if prompt_text is not None:
            input_ids = tokenizer.encode(prompt_text, return_tensors='pt').to(device)
            attention_mask = torch.ones_like(input_ids).to(device)
        else:
            input_ids = torch.tensor([[tokenizer.eos_token_id]], dtype=torch.long).to(device)
            attention_mask = torch.ones_like(input_ids).to(device)

        if image_path is not None:
            try:
                image = Image.open(image_path).convert('RGB')
                image = processor(images=image, return_tensors="pt")['pixel_values']
                if image.dim() == 4 and image.size(0) == 1:
                    image = image.squeeze(0)  # Shape: (C, H, W)
                else:
                    raise ValueError(f"Unexpected pixel_values shape in generate: {image.shape}")
                image = image.to(device)
            except Exception as e:
                print(f"Error processing image: {e}")
                image = None
        else:
            image = None

        with torch.no_grad():
            if image is not None:
                # Add batch dimension
                image = image.unsqueeze(0)  # Shape: (1, C, H, W)
                image_features = model.clip.get_image_features(pixel_values=image)
                adapter_features = model.adapter(image_features)  # Shape: (1, projection_dim)
                # Expand adapter_features to match sequence length
                adapter_features = adapter_features.unsqueeze(1).expand(-1, input_ids.size(1), -1)
                # Get token embeddings
                token_embeddings = model.gpt2.transformer.wte(input_ids)  # LongTensor
                # Combine token embeddings with adapter features
                inputs_embeds = token_embeddings + adapter_features
                # Generate with inputs_embeds
                outputs = model.gpt2.generate(
                    inputs_embeds=inputs_embeds,
                    attention_mask=attention_mask,
                    max_length=50
                )
            else:
                outputs = model.gpt2.generate(
                    input_ids=input_ids,
                    attention_mask=attention_mask,
                    max_length=50
                )

        return tokenizer.decode(outputs[0], skip_special_tokens=True)

    # Examples of generation
    print(generate(
        prompt_text="Question: What do respiration and combustion give out? Choices: A. Oxygen B. Carbon dioxide C. Nitrogen D. Heat Answer with the letter."
    ))
    print(generate(image_path='example.jpg'))
    print(generate(
        prompt_text="Question: What do respiration and combustion give out? Choices: A. Oxygen B. Carbon dioxide C. Nitrogen D. Heat Answer with the letter.",
        image_path='example.jpg'
    ))


if __name__ == "__main__":
    main()