ObjectJEPA

An object-centric world model that forecasts physical scenes in latent space, object by object, and answers "what if" and "what will happen next?" questions.

Built for the Meta Global AI Developer Hackathon (2026), Indie Developer track, "Push the Frontier". Code, evaluation and demos: github.com/idrissa-gif/ObjectJEPA

A frozen V-JEPA 2 encoder turns each object (found by SAM 3) into a latent. The predictor here rolls those latents forward in time, without generating a single pixel. A learned collision head reads collisions out of the forecast, and a learned update turns "remove this object and roll out again" into the answer to a "what if" question. An LLM agent (Muse Spark) calls these as tools.

Files

folder	model	size	format
predictor/	Object predictor. Transformer over one latent per object (8 slots, 1024-d, from V-JEPA 2 ViT-L) and its position. 32 observed frames → the next 32, rolled out step by step. Width 512, depth 8, 60 epochs.	26.3M params	predictor_best.pt: state dict, latent mean/std, config. Loads with torch.load(..., weights_only=True). history.json: training curve.
collision_head/	Collision head. Temporal convolution over pairs of object latents plus geometry → a contact logit per step for each of the 28 object pairs. Threshold 1.5.	0.47M params	head_best.pt, history.json
whatif_update/	What-if update. Gradient-boosted classifier over the head's scores on the real video, the factual rollout and the counterfactual rollout.	300 trees	update.pkl: scikit-learn 1.6 HistGradientBoostingClassifier, pickled. Unpickling runs code: load it only from a source you trust.

The encoder and the segmenter are not included: the predictor works on features of the frozen facebook/vjepa2-vitl-fpc64-256, pooled inside SAM 3 masks (or the CLEVRER masks), both used as they are.

Use

git clone https://github.com/idrissa-gif/ObjectJEPA && cd ObjectJEPA
hf download Idrissa242/ObjectJEPA --local-dir weights

from whatif import load_model
from collision_head import load_head
from whatif_update import load_update

model, mu, sd, cfg = load_model("weights/predictor")
head, thr = load_head("weights/collision_head")
update = load_update("weights/whatif_update")

The agent and the demos take the folders directly: --ckpt weights/predictor --head weights/collision_head --update weights/whatif_update (agent.py, demo_live.py, demo_web.py, visualize.py). Per-object features come from sam_masks.py and extract_object_features.py; the GitHub README has the full pipeline.

Training data

CLEVRER (Yi, Gan, et al., ICLR 2020; CC0): 10,000 training videos of objects moving and colliding. 1,000 of them are held out by video for validation and never seen in training.

• Predictor: L1 loss in normalised latent space plus a position loss, on object features from the CLEVRER masks. It transfers to SAM 3 masks with almost no loss.
• Collision head: CLEVRER's annotated collisions, on both real and predicted sequences (8,000 videos; 1,000 other training videos pick the epoch and threshold).
• What-if update: the counterfactual questions of those 1,000 selection videos, never the validation videos. It sees only world-model signals, not shortcuts about the removed object's own contacts.

Results

On the 1,000 held-out validation videos:

task	ObjectJEPA	baseline
object forecast: latent L1 / position error	0.581 / 4.6 px	0.683 / 16.3 px (mean / constant velocity)
same, on SAM 3 masks (200 videos)	0.589 / 4.3 px
collision F1 on forecast steps	0.898	0.762 (distance rule)
"What will happen next?" (714 questions, chance 0.5)	0.973	0.933 (constant velocity)
"What if X is removed?" (1,858 questions), per option / per question	0.844 / 0.564	0.830 / 0.522 (no change)

Full agent with no ground truth at all (Muse Spark calling the tools, SAM 3 masks, 200 validation videos): 0.966 on "what will happen next?" (148 questions) and 0.811 / 0.517 on "what if" (373 questions).

Limits

• Trained and tested on CLEVRER only: synthetic scenes, up to 8 objects, fixed camera. Not tested on real video.
• On "what if", most of the gain over the first version comes from reading the real video better. The counterfactual rollout itself adds about one point over answering "nothing changes".

Built during the hackathon

These weights were trained during the event (Oct 26 – Nov 4, 2026), with code written during the event. Used as they are, not trained here: V-JEPA 2, SAM 3 and Muse Spark by Meta, and the CLEVRER dataset.

License

MIT. V-JEPA 2, SAM 3 and CLEVRER keep their own licences.