ConCor-1

Vision-Language Grounding as Bidirectional Concept Correspondence Jieyu Zhang*, Ziqi Gao*, Luke Zettlemoyer, Ranjay Krishna (*equal contribution) — University of Washington, Allen Institute for AI, Microsoft SuperIntelligence, FAIR at Meta

ConCor-1 recovers all correspondences between visually referential text spans and instance-level image segments in an image–text pair. Unlike conventional grounding, the text spans are not given as queries: the model decides which parts of the text are visually referential, groups co-referring mentions, segments the corresponding objects, and decides how many correspondences exist.

This single formulation covers phrase grounding, referring-expression grounding and open-vocabulary detection/segmentation — a caption, a referring expression and a category list are all just "the text" for ConCor-1.

How it works

ConCor-1 uses the pretrained Qwen3.5-0.8B vision-language model as a contextual image–text encoder rather than as a generator, and appends 385 learnable bridge tokens to the multimodal sequence:

[<vision_start>, <image_pad> × N_v, <vision_end>, text tokens × N_t, bridge tokens × 385]

Each bridge token represents one candidate image–text correspondence. Three lightweight heads read its final hidden state:

Head	Prediction	Implementation
presence_head	is this a valid correspondence?	SwiGLU MLP → scalar logit
text_segmentation_head	binary mask over the input text tokens	bridge/text features → 256-d correspondence space → bilinear scorer
vision_segmentation_head	binary mask over the image	patch expander (4 MLPs, undoing 2×2 patch merging) + pre-merger ViT feature fusion + 2-block convolutional decoder → 4-px cell grid → bilinear scorer

Bridge tokens are organised into multi-scale spatial groups — grids from 1×1 to 10×10, so Q = Σ s² = 385 — which gives them location and scale priors (coarse levels for large instances, fine levels for small ones). The backbone's full-attention layers run with a bidirectional mask so bridge tokens can access the complete multimodal context, while the linear-attention layers keep Qwen3.5's pretrained behaviour. The model is therefore not autoregressive: use_cache, past_key_values and generate() are not supported.

Parameters (Table 4 of the paper):

Component	Params
Qwen3.5-0.8B backbone	852.99 M
Presence head	0.52 M
Text segmentation head	1.11 M
Vision segmentation head	12.01 M
Bridge-token embeddings	0.145 M
ConCor-1 total	866.79 M (+1.62 %)

Usage

The example below is a real validation sample, bundled as example.png: image 000000000285 from COCO val2017 with its COCONut-PanCap caption.

import torch
from PIL import Image
from transformers import AutoModel, AutoProcessor

model_id = "UWGZQ/ConCor-1"  # or a local path to this repository

processor = AutoProcessor.from_pretrained(model_id, trust_remote_code=True)
model = AutoModel.from_pretrained(
    model_id,
    trust_remote_code=True,
    dtype=torch.bfloat16,
    attn_implementation="flash_attention_2",  # "sdpa" also works
).to("cuda").eval()

image = Image.open("example.png").convert("RGB")
text = (
    "This image depicts a close-up of a brown bear in a natural outdoor setting. "
    "The background consists of lush green grass. In the foreground, a large brown bear "
    "is positioned centrally."
)

inputs = processor(images=image, text=text, return_tensors="pt").to("cuda")
with torch.inference_mode(), torch.autocast("cuda", dtype=torch.bfloat16):
    outputs = model(**inputs)

correspondences = processor.post_process_correspondences(
    outputs, text=text, target_sizes=[(image.height, image.width)]
)[0]

for correspondence in correspondences:
    print(
        f"{correspondence['presence_score']:.3f}",
        correspondence["text_phrases"],   # phrases of this correspondence's text mask
        correspondence["text_spans"],     # character spans into `text`
        correspondence["mask"].shape,     # bool array, (height, width)
    )

Output — two correspondences, which is exactly what this sample is annotated with:

1.000 ['a brown bear', 'a large brown bear']   [[33, 45], [140, 158]]   (640, 586)
0.999 ['green grass']                          [[108, 119]]             (640, 586)

Note the first line: the caption mentions the bear twice, and both mentions land in a single text mask — which is why a text mask is a set of character spans rather than one span. The ground-truth annotation for this sample groups the same two mentions ([[33, 45], [140, 158]]) and pairs them with one bear mask.

example_inference.py wraps this up as a script, including a mask-overlay renderer:

python example_inference.py \
    --image example.png \
    --text "This image depicts a close-up of a brown bear in a natural outdoor setting. The background consists of lush green grass. In the foreground, a large brown bear is positioned centrally." \
    --output overlay.png

Inference thresholds

Post-processing follows the paper: keep bridge tokens above a presence threshold, threshold the text and image mask probabilities, then remove duplicate correspondences with NMS (a candidate is suppressed only when both its image-mask IoU and its text-span IoU with a higher-scoring candidate exceed the threshold).

Threshold	Default	Argument
presence	0.10	presence_threshold
text mask	0.45	text_threshold
image mask	0.45	image_threshold
NMS IoU	0.50	nms_iou_threshold

Image masks are produced by reshaping the per-bridge logits onto their 4-px cell grid, bilinearly upsampling in logit space to target_sizes, and then applying the sigmoid and threshold.

Other input modes

# A category list instead of a caption (open-vocabulary detection/segmentation).
inputs = processor(images=image, text="bear . grass . tree . person", return_tensors="pt")
# -> 0.995 ['bear'] [[0, 4]] ; 0.980 ['grass'] [[7, 12]]
#    the absent categories ("tree", "person") produce no correspondence

# A batch of image-text pairs (right-padded internally).
inputs = processor(images=[image_a, image_b], text=[text_a, text_b], return_tensors="pt")
results = processor.post_process_correspondences(
    outputs, text=[text_a, text_b],
    target_sizes=[(image_a.height, image_a.width), (image_b.height, image_b.width)],
)

# Text only: which spans are visually referential, and how do they group?
inputs = processor(images=None, text=text, return_tensors="pt")   # image_mask_logits is None

Raw model outputs

model(**inputs) returns a ConCor1Output:

Field	Shape	Meaning
presence_logits	(B, 385)	correspondence presence
text_mask_logits	(B, 385, N_text)	per-bridge mask over the text tokens
image_mask_logits	(B, 385, N_cells)	per-bridge mask over image cells, row-major
image_mask_grid_hw	(B, 2)	(height, width) of each sample's cell grid

So image_mask_logits[b, q, : h * w].reshape(h, w) is bridge q's mask map, where one cell covers 4 px of the processed image.

Training and evaluation data

Training converts grounding, segmentation and referring-expression datasets into a unified correspondence format (image masks + text masks + correspondence labels), using an LLM rewriting pipeline that groups co-referring mentions and drops mentions without a visual referent: GoldG (Flickr30k + GQA, pseudo-masks from SAM3), COCONut-PanCap, GroundedRef-train (LLM-written referring/compositional captions over COCO panoptic), plus COCO, COCONut, LVIS, ADE20K instance segmentation and the counting subset of PixMo-Point — 678 k caption-style images and 709 k instance-segmentation images.

Evaluation uses Flickr30k (4 k pairs), COCONut-PanCap (3 k) and GroundedRef (2 k), scored along three axes — text segmentation, image segmentation and joint correspondence precision/recall/F1. See the paper for the numbers, ablations and baselines.

This checkpoint is the released main model: 100 k steps, 385 bridge tokens, a fixed ~1.0 M-pixel image budget (min_pixels = max_pixels = 1003520, i.e. 1024 visual tokens).

Files

File	Purpose
configuration_concor1.py	ConCor1Config
modeling_concor1.py	ConCor1ForConceptCorrespondence and its heads
processing_concor1.py	ConCor1Processor: sequence construction + correspondence post-processing
example_inference.py	image + text → correspondences, with mask overlay
example.png	the example image: COCO val2017/000000000285, from the COCONut-PanCap validation split
convert_original_checkpoint.py	converts a research training checkpoint into this repository
model.safetensors	weights: bf16 backbone, fp32 prediction heads

The prediction heads are stored and loaded in fp32 (via _keep_in_fp32_modules_strict) exactly as the training checkpoint holds them, while the backbone is bf16. Every head casts its inputs to its own parameter dtype, so the model runs with or without torch.autocast. With attn_implementation="flash_attention_2" under torch.autocast(bfloat16), this repository reproduces the research code's presence, text-mask and image-mask logits bit-for-bit (verified single-sample and batched). sdpa and running without autocast stay within bf16 kernel noise (binary-mask IoU ≈ 0.98–0.995 against the exact path).

Requirements

torch >= 2.8
transformers >= 5.3        # needs the Qwen3.5 architecture
flash-attn >= 2.8          # optional; use attn_implementation="sdpa" instead
fla / causal-conv1d        # Qwen3.5's gated-delta-net kernels

Limitations

Mask quality remains behind specialized detection/segmentation models — the image-mask head is deliberately lightweight and optimised for joint correspondence prediction rather than segmentation quality alone. The model depends on correspondence-style supervision that had to be converted from datasets not natively annotated that way, which introduces noise. Grounding errors can propagate to downstream systems, and performance may be uneven across categories, visual domains and languages, so validate carefully before using it in human-facing or safety-critical settings.

Demo

An interactive ZeroGPU demo lives at UWGZQ/ConCor-1-demo.

Citation

@article{zhang2026concor,
  title   = {Vision-Language Grounding as Bidirectional Concept Correspondence},
  author  = {Zhang, Jieyu and Gao, Ziqi and Zettlemoyer, Luke and Krishna, Ranjay},
  year    = {2026}
}

The backbone is Qwen3.5-0.8B (Apache 2.0).