Update README.md

README.md

@@ -3,165 +3,10 @@ license: apache-2.0
 language:
 - en
 datasets:
-- HuggingFaceM4/WebSight
+- HuggingFaceM4/RAVEN
 tags:
 - code
 ---
 
 
-# Model Description
-
-This model converts screenshots of website components into HTML/Tailwind CSS codes.
-
-It is based on an early checkpoint of our forthcoming vision-language foundation model, which has been further fine-tuned with DoRA using the [Websight-v1](https://huggingface.co/datasets/HuggingFaceM4/Websight) dataset.
-
-The base model is built upon Mistral-7B and SigLIP-SO400M, and uses the Patch n’ Pack strategy to preserve the original aspect ratio of the input images, with a resolution of up to 980 pixels for each side.
-Further insights into the model’s architecture and its training process will be detailed upon its release.
-
-The goal of open-sourcing the WebSight dataset along with the model Sightseer is to kick off an effort to develop improved models capable of converting a website screenshot into actual code.
-
-Try out the [demo](https://huggingface.co/spaces/HuggingFaceM4/screenshot2html)!
-
-
-# Code snippet
-
-The code snippet demonstrates how to perform batched generation to convert screenshots of websites into corresponding HTML + Tailwind code.
-
-Note that the logic to process, and pad inputs will be encapsulated into a user-friendly processor upon the release of our vision and language model.
-
-```python
-import torch
-import requests
-
-from datasets import load_dataset
-from io import BytesIO
-from transformers import AutoModelForCausalLM, AutoProcessor
-from PIL import Image
-
-from transformers.image_utils import to_numpy_array, PILImageResampling, ChannelDimension
-from transformers.image_transforms import resize, to_channel_dimension_format
-
-
-DEVICE = torch.device("cuda")
-PROCESSOR = AutoProcessor.from_pretrained(
-    "HuggingFaceM4/Sightseer",
-    token=API_TOKEN,
-)
-MODEL = AutoModelForCausalLM.from_pretrained(
-    "HuggingFaceM4/Sightseer",
-    token=API_TOKEN,
-    trust_remote_code=True,
-    torch_dtype=torch.bfloat16,
-).to(DEVICE)
-image_seq_len = MODEL.config.perceiver_config.resampler_n_latents
-BOS_TOKEN = PROCESSOR.tokenizer.bos_token
-BAD_WORDS_IDS = PROCESSOR.tokenizer(["<image>", "<fake_token_around_image>"], add_special_tokens=False).input_ids
-
-
-def convert_to_rgb(image):
-    if image.mode == "RGB":
-        return image
-
-    image_rgba = image.convert("RGBA")
-    background = Image.new("RGBA", image_rgba.size, (255, 255, 255))
-    alpha_composite = Image.alpha_composite(background, image_rgba)
-    alpha_composite = alpha_composite.convert("RGB")
-    return alpha_composite
-
-
-# The processor is the same as the Idefics processor except for the BILINEAR interpolation,
-# so this is a hack in order to redefine ONLY the transform method
-def custom_transform(x):
-    x = convert_to_rgb(x)
-    x = to_numpy_array(x)
-
-    height, width = x.shape[:2]
-    aspect_ratio = width / height
-    if width >= height and width > 980:
-        width = 980
-        height = int(width / aspect_ratio)
-    elif height > width and height > 980:
-        height = 980
-        width = int(height * aspect_ratio)
-    width = max(width, 378)
-    height = max(height, 378)
-
-    x = resize(x, (height, width), resample=PILImageResampling.BILINEAR)
-    x = PROCESSOR.image_processor.rescale(x, scale=1 / 255)
-    x = PROCESSOR.image_processor.normalize(
-        x,
-        mean=PROCESSOR.image_processor.image_mean,
-        std=PROCESSOR.image_processor.image_std
-    )
-    x = to_channel_dimension_format(x, ChannelDimension.FIRST)
-    x = torch.tensor(x)
-    return x
-
-
-# Create text token inputs
-image_seq = '<image>' * image_seq_len
-inputs = PROCESSOR.tokenizer(
-    [
-        f"{BOS_TOKEN}<fake_token_around_image>{image_seq}<fake_token_around_image>In this image, we see",
-        f"{BOS_TOKEN}bla bla<fake_token_around_image>{image_seq}<fake_token_around_image>{image_seq}<fake_token_around_image>",
-    ],
-    return_tensors="pt",
-    add_special_tokens=False,
-    padding=True,
-)
-
-
-# Create pixel inputs
-# We load images from WebSight, but any screenshot in the form of a PIL image will work
-dataset = load_dataset("HuggingFaceM4/WebSight", streaming=True)
-dataset = iter(dataset)
-image1 = next(dataset)
-image2 = next(dataset)
-raw_images = [
-    [image1],
-    [image2],
-]
-output_images = [
-    [PROCESSOR.image_processor(img, transform=custom_transform) for img in img_list]
-    for img_list in raw_images
-]
-total_batch_size = len(output_images)
-max_num_images = max([len(img_l) for img_l in output_images])
-max_height = max([i.size(2) for img_l in output_images for i in img_l])
-max_width = max([i.size(3) for img_l in output_images for i in img_l])
-padded_image_tensor = torch.zeros(total_batch_size, max_num_images, 3, max_height, max_width)
-padded_pixel_attention_masks = torch.zeros(
-    total_batch_size, max_num_images, max_height, max_width, dtype=torch.bool
-)
-for batch_idx, img_l in enumerate(output_images):
-    for img_idx, img in enumerate(img_l):
-        im_height, im_width = img.size()[2:]
-        padded_image_tensor[batch_idx, img_idx, :, :im_height, :im_width] = img
-        padded_pixel_attention_masks[batch_idx, img_idx, :im_height, :im_width] = True
-
-inputs["pixel_values"] = padded_image_tensor
-inputs["pixel_attention_mask"] = padded_pixel_attention_masks
-inputs = {k: v.to(DEVICE) for k, v in inputs.items()}
-
-generated_ids = MODEL.generate(**inputs, bad_words_ids=BAD_WORDS_IDS, max_new_tokens=10)
-generated_texts = PROCESSOR.batch_decode(generated_ids, skip_special_tokens=True)
-
-print(generated_texts)
-```
-
-
-# Model Details
-
-- **Developed by:** Hugging Face
-- **Model type:** Multi-modal model (screenshot of website component to HTML/Tailwind CSS code)
-- **Language(s) (NLP):** en
-- **License:** see [License section](#license)
-- **Parent Models:** [SigLIP](https://github.com/huggingface/transformers/pull/26522) and [mistralai/Mistral-7B-v0.1](https://huggingface.co/mistralai/Mistral-7B-v0.1)
-- **Resources for more information:**
-    - WebSight dataset: [Dataset card](https://huggingface.co/datasets/HuggingFaceM4/Websight)
-
-# License
-
-The model is built on top of two pre-trained models: [SigLIP](https://github.com/huggingface/transformers/pull/26522) and [mistralai/Mistral-7B-v0.1](https://huggingface.co/mistralai/Mistral-7B-v0.1), which are delivered under an Apache-2.0 license. As such, users should comply with the licenses of these models.
-
-The two pre-trained models are connected to each other with newly initialized parameters that we train. These are not based on any of the two base frozen models forming the composite model. We release the additional weights we trained under an Apache-2.0 license.
+idefics2 (upcoming) fine-tuned on RAVEN problems