gemma-3-27b-it-vlm-react-screenshot-to-code

A LoRA adapter for google/gemma-3-27b-it (Gemma 3 27B IT, vision-language) that turns a screenshot of a UI into React component code.

Give it an image of an interface; it returns a single self-contained React component that reproduces the layout.

This is the 27B sibling of gemma-3-4b-it-vlm-react-screenshot-to-code.

👉 Live demo: Reubencf/react-screenshot-to-code

Model details

Base model	google/gemma-3-27b-it (Gemma3ForConditionalGeneration)
Adapter type	LoRA (PEFT 0.15.1)
Rank / alpha	r=4, lora_alpha=8, lora_dropout=0.0
Target modules	q_proj, v_proj
Adapted layers	Language model self-attention only, all 62 decoder layers (248 tensors)
Frozen	Full SigLIP vision tower and multimodal projector (listed in exclude_modules)
Adapter size	~16 MB (float32)
Base precision	bfloat16

Only the language model's query and value projections are adapted — the vision encoder is untouched, so the adapter changes how the model writes code, not how it sees images.

The rank is lower than the 4B adapter's (r=4 against r=16) and it touches two projections rather than four, so it is a lighter intervention on a much larger base.

Note on the base model. The training run recorded its base as togethercomputer/gemma-3-27b-it-VLM, which is not a resolvable Hub repo (404). The bundled config.json matches google/gemma-3-27b-it exactly (hidden 5376, 62 text layers, 32 query heads / 16 KV heads, SigLIP-27 @ 896px, vocab 262208), and the adapter's own tensor shapes agree (q_proj 5376→4096, v_proj 5376→2048), so base_model_name_or_path has been corrected to point there. unsloth/gemma-3-27b-it is an ungated mirror of the same model if you would rather not request access.

Training

Short LoRA run, recorded in trainer_state.json:

Epochs	1
Steps	26 (max_steps=26)
Train batch size	1
Logging / eval every	1 / 5 steps
Total FLOPs	1.03e18
Start train loss	0.624
Final eval loss	0.590

Eval loss fell at every checkpoint (0.631 → 0.617 → 0.603 → 0.593 → 0.590), so the run was still improving when it stopped.

This is a light adaptation, not a full fine-tune — 26 steps at batch size 1. Expect it to nudge output format and style toward React rather than to teach new visual reasoning. Treat generations as a starting scaffold to edit, not production code.

Usage

Requires transformers >= 5.10.1 — the adapter keys use the refactored model.language_model.layers.* module layout and will not match older checkpoints.

The base model is ~54 GB in bfloat16. On a single 24 GB card, load it 4-bit and skip the merge (see below).

import torch
from transformers import AutoProcessor, Gemma3ForConditionalGeneration
from peft import PeftModel
from PIL import Image

BASE = "google/gemma-3-27b-it"          # or "unsloth/gemma-3-27b-it" (ungated mirror)
ADAPTER = "Reubencf/gemma-3-27b-it-vlm-react-screenshot-to-code"

processor = AutoProcessor.from_pretrained(BASE)
model = Gemma3ForConditionalGeneration.from_pretrained(
    BASE, dtype=torch.bfloat16, device_map="auto"
)
model = PeftModel.from_pretrained(model, ADAPTER).merge_and_unload().eval()

image = Image.open("screenshot.png").convert("RGB")
messages = [{
    "role": "user",
    "content": [
        {"type": "image", "image": image},
        {"type": "text", "text":
         "Convert this screenshot into a single self-contained React component. "
         "Use Tailwind CSS classes for styling. Return only code."},
    ],
}]

inputs = processor.apply_chat_template(
    messages, add_generation_prompt=True, tokenize=True,
    return_dict=True, return_tensors="pt",
).to(model.device, dtype=torch.bfloat16)

with torch.inference_mode():
    out = model.generate(**inputs, max_new_tokens=1536, do_sample=True,
                         temperature=0.7, top_p=0.95)

print(processor.decode(out[0][inputs["input_ids"].shape[-1]:], skip_special_tokens=True))

Fitting on one 24 GB GPU

Load the base in NF4 and leave the adapter unmerged — merge_and_unload() folds the LoRA deltas back into the base weights at their original precision, which a quantised base cannot represent.

from transformers import BitsAndBytesConfig

quant = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype=torch.bfloat16,
    bnb_4bit_use_double_quant=True,
)
model = Gemma3ForConditionalGeneration.from_pretrained(
    BASE, quantization_config=quant, device_map="auto"
)
model = PeftModel.from_pretrained(model, ADAPTER).eval()   # note: no merge

Prompt format

Standard Gemma 3 chat template (chat_template.jinja is included). Images are inserted as <start_of_image> and expand to 256 soft tokens each at 896×896.

<bos><start_of_turn>user
<start_of_image>Convert this screenshot into a React component.<end_of_turn>
<start_of_turn>model

Limitations

• 26 training steps. Output quality is closer to the base model than to a task-specialised one.
• Single-screenshot, single-component. No routing, state management, or multi-page flows.
• Colours, spacing, and font sizes are approximated from the image — expect to tune them.
• Text in the screenshot is transcribed by the vision encoder and can be misread at small sizes.
• Generated code is unverified. Review it before running, as you would any model output.
• Inherits all limitations and the licence of the Gemma 3 base model.

Licence

Governed by the Gemma Terms of Use. The adapter weights are a derivative of Gemma 3 and are subject to the same terms.

Framework versions

• PEFT 0.15.1
• Transformers 5.10.1