MoistPotato/precode-jepa-qwen2.5-coder-1.5b

PreCode-JEPA checkpoints (Qwen2.5-Coder-1.5B)

HashCode 14.0, Track 02 (team 21Squad). Code: https://github.com/hackHashCode/v1 (branch nikhita/studio, see REPORT.md).

PreCode trains a code model to predict an embedding of what its program will do (the reference solution's execution trace on probe inputs, function renamed to f) from the task description, alongside next-token prediction.

folder	method	MBPP+ pass@1 (5-seed mean)	HumanEval+
precodefull_aux_s<seed>	PreCode (behaviour objective; includes pred_head.pt)	60.0	18.0
sft_s<seed>	SFT	55.0	16.8
sft_loop_s<seed>	SFT control (PreCode's loop, behaviour loss off)	54.7	16.2
jepa_s<seed>	LLM-JEPA	46.9	26.1

Each folder is a merged full model (no adapter needed). Prompt format: ChatML with system prompt "You are an expert Python programmer. Write a correct Python function for the task. Output only code." and the task description plus its first assert as the user turn; greedy decoding; stop at any <|...|> control token.

from transformers import AutoModelForCausalLM, AutoTokenizer
repo, run = "MoistPotato/precode-jepa-qwen2.5-coder-1.5b", "precodefull_aux_s42"
tok = AutoTokenizer.from_pretrained(repo, subfolder=run)
model = AutoModelForCausalLM.from_pretrained(repo, subfolder=run, torch_dtype="auto")