fable-glm-vision

A 44.8M-parameter vision projector that gives GLM-5.2 image understanding — trained with the frozen-backbone recipe pioneered by Harry Partridge (@part_harry_) in his "GLM 5.2 With Vision" write-up, which is the original source of this method:

"It turns out that the answer to this question is yes. Amazingly, by training only a tiny 2 layer MLP with just 50M parameters, we are able to give GLM 5.2 vision capabilities equivalent to Claude 4.5 Haiku on MMMU-Pro (55%)."

"If you are interested in using GLM 5.2 with vision, let me know in the comments or send me a message!"

— Harry Partridge, original post

All credit for the method — frozen vision tower + frozen LLM, train only the projector, watch for the SFT grokking event — goes to Harry Partridge. This repo is an independent reproduction/extension focused on computer-use grounding (UI screenshots → click coordinates) rather than general VQA.

Architecture

image → MoonViT (Kimi K2.6 vision tower, frozen, ~450M)
      → VisionProjector (THIS REPO, 44.8M, the only trained part)
      → GLM-5.2 (frozen)

The projector is a 2-layer MLP:

tensor	shape
ln.{weight,bias}	LayerNorm(1152)
fc1.{weight,bias}	Linear(1152 → 6144) + GELU
fc2.{weight,bias}	Linear(6144 → 6144)

Projected vectors replace the <|image|> placeholder tokens (id 154854) in GLM's embedding stream, inside a <|begin_of_image|> … <|end_of_image|> span. Coordinates are expressed as click(start_box=[x,y]) with x, y normalized to [0, 999].

This checkpoint

• Trained on UI screenshot grounding: wave-ui-25k + ShowUI-desktop (30.8k examples, screenshot + instruction → click coordinate), batch 64, lr 5e-4, frozen NF4-quantized GLM-5.2 backbone (32/256 experts).
• Status: work-in-progress, pre-grokking checkpoint (~800 cumulative optimizer steps). The characteristic sharp loss drop has not occurred yet; spatial-feature probes already show ~2000× above-random variance. Expect this repo to be updated as training progresses.
• Sibling runs in progress: multistep browser/OS trajectories (Mind2Web, AITW, MiniWob, GUIAct) and art/visual understanding (WikiArt, product design). Checkpoint archive: 0xSero/glm-vision-checkpoints.

Usage sketch

import torch, torch.nn as nn
from safetensors.torch import load_file

class VisionProjector(nn.Module):
    def __init__(self):
        super().__init__()
        self.ln = nn.LayerNorm(1152)
        self.fc1 = nn.Linear(1152, 6144)
        self.fc2 = nn.Linear(6144, 6144)
    def forward(self, x):           # x: (B, N_patches, 1152) from MoonViT
        return self.fc2(torch.nn.functional.gelu(self.fc1(self.ln(x))))

proj = VisionProjector()
proj.load_state_dict(load_file("projector.safetensors"))
# encode image with MoonViT (Kimi K2.6 vision tower), project, then splice
# the outputs into GLM-5.2's input embeddings at the <|image|> positions.

You need GLM-5.2 and the Kimi K2.6 vision tower (MoonViT) separately — this repo is only the 44.8M projector that connects them.