bmeivar/kap-aperture-lora

KAP aperture-conditioned LoRA adapters

Rank-8 LoRA adapters from "Object-Only Is Not Context-Free: A Counterfactual Diagnostic for Context Removal in Object Retrieval" (Boaz Meivar, Shai Avidan; WACV 2027). Each adapter makes a backbone's global descriptor stay stable as the visible context around an object is removed.

folder	timm base model	descriptor	LoRA targets	trainable params
vit_l	vit_large_patch16_384.augreg_in21k_ft_in1k	CLS token	attn.qkv	0.26%
siglip2_l	vit_large_patch16_siglip_384.v2_webli	attention-pool head (pre_logits)	attn.qkv	0.25%
eva02_mim	eva02_large_patch14_224.mim_in22k	mean pool (pre_logits)	attn.q_proj/k_proj/v_proj	0.39%

Training

Teacher-student aperture distillation on one backbone: the teacher (LoRA off, gradient stopped) sees a segment through a wide aperture (d ~ U(20, 250) px), the student (LoRA on) through a tighter one, with the outside filled by gray, zero, blur or noise. The loss is InfoNCE with in-batch negatives (temperature 0.07). Settings: 1 epoch over 25K COCO train2017 images, batch 16, AdamW (lr 2e-4, weight decay 0.01), rank 8, alpha 16, dropout 0.1, seed 43. Training uses train2017 only; evaluation uses COCO val2017, INSTRE and ILIAS.

Usage

import timm
from peft import PeftModel

base = timm.create_model("vit_large_patch16_384.augreg_in21k_ft_in1k", pretrained=True, num_classes=0)
model = PeftModel.from_pretrained(base, "<local path>/vit_l")

With the KAP code, download this repository and pass a folder as --adapter, e.g. python -m kap.run --backbone vit_l --adapter kap-aperture-lora/vit_l ....

Citation

@inproceedings{meivar2027kap,
  title     = {Object-Only Is Not Context-Free: A Counterfactual Diagnostic for Context Removal in Object Retrieval},
  author    = {Meivar, Boaz and Avidan, Shai},
  booktitle = {Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV)},
  year      = {2027}
}