LINK: Learning Instance-level Knowledge from Vision-Language Models for Human-Object Interaction Detection
Authors: WU EASTMAN Z Y
Ya-Li Li
Yuan Wang
Shengjin Wang
Venue: ICLR 2026 Poster (2026)
Keywords: Human-object interaction detection
Vision-language model
zero-shot
open-vocabulary
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning

Abstract: Human-Object Interaction (HOI) detection with vision-language models (VLMs) has progressed rapidly, yet a trade-off persists between specialization and generalization. 
Two major challenges remain: (1) the sparsity of supervision, which hampers effective transfer of foundation models to HOI tasks, and (2) the absence of a generalizable architecture that can excel in both fully supervised and zero-shot scenarios. 
To address these issues, we propose \textbf{LINK}, \textbf{L}earning \textbf{IN}stance-level \textbf{K}nowledge from VLMs. 
First, we introduce a HOI detection framework equipped with a Human-Object Geometrical Encoder and a VLM Linking Decoder. 
By decoupling from detector-specific features, our design ensures plug-and-play compatibility with arbitrary object detectors and consistent adaptability across diverse settings. 
Building on this foundation, we develop a Progressive Learning Strategy under a teacher-student paradigm, which delivers dense supervision over all potential human-object pairs. 
By contrasting subtle spatial and semantic differences between positive and negative instances, the model learns robust and transferable HOI representations. 
LINK sets new state-of-the-art on SWiG-HOI, HICO-DET, and V-COCO across zero-shot, fully supervised, and open-vocabulary settings, with strong generalization to unseen interaction categories.

Published as a conference paper at ICLR 2026
LINK: LEARNING INSTANCE-LEVEL KNOWLEDGE FROM VISION-LANGUAGE MODELS FOR HUMAN- OBJECT INTERACTION DETECTION
Eastman Z Y, Wu Yali Li Yuan Wang1.2.5, Shengjin Wang1.234
Department of Electronic Engineering, Tsinghua University
2 Beijing National Research Center for Information Science and Technology (BNRist), China
3 National Engineering Research Center Dangerous Articles and Explosives Detection Technologies, Beijing 100084, China
2yu- t..3:emails..ssshhuuuu tsinghuu edu. en [liyali13, w edu.
ABSTRACT
Human-Object Interaction (HOI) detection with vision-language models (VLMs) has progressed rapidly, yet trade-off persists between specialization and general- ization. Two major challenges remain: (1) the sparsity of supervision, which ham- pers effective transfer of foundation models to HOI tasks, and (2) the absence of a generalizable architecture that can excel in both fully supervised and zero-shot scenarios. To address these issues, we propose LINK, Learning INstance-level Knowledge from VLMs. First, we introduce a HOI detection framework equipped with Human- Object Geometrical Encoder and a VLM Linking Decoder. By de- coupling from detector-specific features, our design ensures plug-and-pla com- patibility with arbitrary object detectors and consistent adaptability across diverse settings. Building on this foundation, we develop Progressive Learning Strategy under a teacher-student paradigm, which delivers dense supervision over all po- tential human-object pairs. By contrasting subtle spatial and semantic differences between positive and negative instances, the model learns robust and transfcrable HOI representations. LINK sets new state-of-the art on SWIG-HOI, HICO DET, and V-COCO across zero-shot, fully supervised, and vocabbacyy settings, with strong generalization unseen interaction categories.
INTRODUCTION
Human-Object Interaction (HOI) detection has recently emerged as a rapidly developing field, re- quiring higher-level visual understanding beyond standard object detection. By focusing on complex human-centric interactions, HOI detection is essential for applications such as intelligent robotics and anomalous behavior detection (Liu et al., 2018). Its core goal is localize human-object pairs and recognize their interactions as structured triplets: < human, action, object >
Recently, foundation models pretrained on large-scale multimodal datasets have shown strong ca- pabilities to provide effective feature representations for downstream tasks. In the field of HOI detection, numerous studies (Lei et al., 2023; Cao et al., 2024; Ning et al., 2023; Mao et al., 2023; Lei et al., 2025b; Wu et al., 2023) have successfully used pretrained CLIPP models (Radford et al., 2021) to improve the recognition of rare and unseen interactions, thereby advancing zero-shot and few-shot learning in HOI detection.
However, existing VLM-based HOI detectors often face an inherent trade-off between specializa- tion and generalization. Dedicated architectures are typically optimized for fully supervised bench- marks, yielding strong in-domain performance but struggling in zero-shot and cross-domain settings due limited generalization capacity. Conversely, zero-shotoriented methods are commonly built upon CLIP with lightweight modifications. While effective in recognizing novel HOI categories,
"Corespooding author.

---

Published as a conference paper at ICLR 2026
(1) Var-l
(2) ViT-base
(1) Object Detection from DETR
(2) Sparse HOI anmotations
Parsozi
Pesson
24
Couchi
Couch:
Poal-hold V
V hotd
book
Cell plane
34
Unlabeied
Inkerazih
(b) Sparse Supervision in HOI Tasks, Although the scene contains diverse interactions, only the red pairs (eg. <personi, lie on, couchs)), are amotated as ground-truth, while valid unlabeled pairs (e.g. "person,, readhold, books)) and informative negatives are ignored.
(a) Zero vo. Fully Supervised:: A Trude-off Perspective
Figure I: Illustration of challenges in HOI detection.
their limited task task-specific adaptability leads to suboptimal performance under fully supervised eval- uation. As illustrated in Figure I(a), clear trade-off emerges: methods achieving higher fully supervised performance often suffer decline in their corresponding zero-shot performance. A sec- ond challenge lies in the sparsity of supervision, as shown in Figure 1(b). In visual scenes, humans and objects form densely connected graph structures, yet ground-truth annotations cover only a small subset of edges, leaving most instances under-utilized. These ignored cases include not only valid but unlabeled pairs, but also informative negatives that could contribute to more robust learn- ing. Together, these issues highlight the core difficulty of adapting VLMs, pretrained on large-scale imagetext pairs, to instance-level HOI detection under sparse supervision.
To address these challenges, we propose LINK, Learning INstance-level Knowledge, which inte- grates architectural innovations with progressive learning strategy. First, we design a HOI detector that introduces a Human-Objer Geometrical Encoder to capture the spatial relationships of paired homan-object instances, and VLM Linking Decoder that bridges VLMs with HOI detection by transforming global semantic representations into fine fine-grained, instance-leve HOI patterns. Second, we develop Progressive Learning Strategy. In the first stage, the model is trained with standard su- pervision to adapt intrinsic knowledge within VLM into HOI-specific patterns. In the second stage, we leverage this pre-trained model as the teacher, where the student receives sparse GT supervision while all human-object pairs, including negatives, are further guided by dense distillation losses. By contrasting subtle spatial and semantic differences between positive and negative instances, the model learns to resolve ambiguities and acquire more discriminative HOI representations.
Our LINK, offers key advantages. It maintains consistency across diverse scenarios (fully super- vised, zero-shot, and open-vocabllary) and captures fine-grained spatial and semantic patterns for robust HOI prediction. Even in cross-domain transfer to synthetic images with drastic semantic shifts, preserved spatial patterns support reliable decisions. In summary, our main contributions are:
We propose a HOI detector with a Human- Objecc Geometrical Encoder and a VLM Link- ing Decoder. This design strengthens HOI-specific reasoning capacity while avoiding un- necessary complexity that may compromise generalization. By decoupling from detector- specific features, LINK achieves plug-and-play compatibility with arbitrary object detec- tors without fine-tuning.
We introduce a progressive learning strategy that delivers dense supervision to all candi- date human-object pairs, enabling the model capture fine-grained spatial and semantic distinctions between positive and negative instances. This effectively mitigates the super- vision sparsity inherent in HOI tasks.
We conduct the first comprehensive evaluation of HOI detection across diverse foundation models (CLIP, BLIP, DIÑOv2, DINO@448, SigLIP2, Florence2), and demonstrate that +LINK consistently improves all baselines, with the most substantial gains on long-tail HOIs (< 10 samples).
Our method outperforms existing methods by a large margin. For instance, on HICO-DET with an R50 backbone and ViT-L CLIP, LINK achieves 42.92 45.03 mAP the full rare sets, surpassing the previous state-of-ther the-art by 3.87/ 6.37 mAP, corresponding to relative gains of 9.9% 16.5%. %. Moreover, when scaled up Swin-L. backbone, LINK further improves to 49.06 / 53.63 mAP.
2

---

Published as a conference paper at ICLR 2026
2 RELATED WORKS
Sluman-Objece Interaction Detection: Human-Object Interaction (HOI) detection is composite task that involves localizing humans and objects, as well as recognizing their interactions. Existing methods can be broadly classified into one-stage and two-stage paradigms. One-stage methods (Liao et al., 2020; Chen & Yanai, 2021; Chen et al., 2021; Kim et al., 2021; Zou al., 2021) aim to jointly localize objects and infer interactions in single forward pass. Early approaches, such as PPDM (Liao et al., 2020) and UnionDet (Kim et al., 2020a), leverage interaction points or union regions as anchors to guide localization and feature extraction. More recently, Transformer-based archi- tectures have advanced the field by introducing query-based HOI detectors. In contrast, two-stage methods (Chao et al., 2018; Gao et al., 2020; 2018; Gkioxari et al., 2018; Gupta et al., 2019; Kim et al., 2020b; Zhou & Chi, 2019; Liu et al., 2020; Wu et al., 2024) decouple the process into object detection followed by HOI classification for each human-object pair. This separation offers greater flexibility, interpretability, and modularity, and has gained increasing attention in recent work. Given the modular nature of the two-stage paradigm, it is particularly well-suited for designing generaliz- able and scalable HOI detectors. In this paper, we aim to develop a unified two-stage HOI detector that performs effectively across both specific benchmarks and generalization scenarios.
Adapting Vision-Languaage Models: The rapid advancement of vision-languee models (VLMs) (Radford et al., 2021; Li al., 2023a; 2022: Wang et al., 2021b: Zhang et al., 2022b) has recently demonstrated strong zero-shot capabilities. This has prompted growing interest in adapting VLMMs for Human-Object Interaction (HOI) detection. For instance, HOICLIP (Ning et al., 2023) em- ploys query-based approach to harness visual knowledge from CLIP, achieving zero-shot HOI enhancement by leveraging CLIPs image-text retrieval capabilities. BCOM (Wang et al., 2024) pro- poses an occlusion-aware Contextual Mining method that guides the model to recover spatial details from occluded feature maps, thereby improving robustness in crowded scenes. ADA-CM employs Concept- -gidedd Memory to retrieve both comain--seecffi and domain domain-agnosti agnostic knowledge from CLIP, enabling a quick adaptation to datasets. More recently, CMMP (Lei et al., 2025b) introduces conditional multi-modal prompts enriched with priors decouple visual representation and interac- tion classification, thereby enhancing the zero-shot HOI detection capability of CLIP-based models.
METHOD
In this section, we first review the problem formulation in VLM-based two-stage HOI detection methods (3.1). We then introduce our proposed unified HOI architecture (3.2), which consists of Human-Object Geometrical Encoder and a VLM Linking Decoder. Next, we present our Progressive Learning Strategy (3.3).
3.1 PRELIMINARY: PROBLEM FORMULATION.
The architecture a two-stage VLM-based HOI detection framework is illustrated in Figure 2. In this paradigm, off-the-shelf object detector is first employed to localize entities. Following standard practice, DETR is used to generate all bounding boxes B, which are divided into human boxes B, and object boxes B.. For each box, corresponding featureeither object queries in DETR or ROI-aligned featuresis extracted and used as a unary query for subsequent interaction reasoning. This results in the mappings Qn ++ B₁, and Q. ++ B.. By enumerating all possible human-object pairs, we construct Qh.o: where each query corresponds to a pair of boxes [B,, 13]-
We then refine Qio via decoder-decoder architecture as Qh.o where F. denotes external features, such as feature maps extracted from CLIP or backbone. The output interaction logits are predicted by Lh FFN(Qn.). As result, for each homan-object pair, the model outputs a prediction in the form of (B), Livo. B,. Co), where C. is the object cate- gory. In the practical training process, supervision is applied only to valid queries @""those where both the human and object bounding boxes have an Intersection over Union (IoU) above 0.5 with ground-tre -trnt boxes. Formally, matched queries are denoted as M and defined as:
Secooder(Encoder(Qn...)
M {Qho (BU B 2 0.5 TOU BB) 2 0.5)
3

---

Published as a conference paper at ICLR 2026
Sec. 3.2 Human Object Geometrical Encoder
Sec. 31 Blaseline
VLM Linking Decoder
Objeet:
Trput Bounding Rooos
HumannObjcct Query
Fenture Map
Buncs S Posiion Encoding
Object Deteciar
Vistun Adgggtrr
Butloaok
Pairwise Geometee Rehiiunship
Zor Allen
Spatial Cress-Atta Iotion spart
Semsntic
Cross
(aRmiLi (ieectod vicle
HOKigumerial Encoder
rebuulllc ic atc
Corest
Solt-Atta
Linking
CLLL
Decodlr
CJ IP-Initasiree Projector
sextar barshell het, Avid hanchall welill bewehal
MIPF
Spersso 145, 1eg .axingg boid
wiold baschall
Figure 2: Overview of the proposed LINK framework. (Sec. 3.1) Baseline: a VLM-based two- stage HOI detection pipeline. (Sec. 3.2) Human-Oby -bject Geometrical Encoder: integrates ROI fea- tures, bounding-box encodings, and pairwise geometry to encode human-object spatial dependen- cies. (Sec. 3.3) VLM Linking Decoder: refines pairwise queries via geometry- and semantic cross-attention, and predicts HOI triplets with a CLIP-initialized head.
Where Ba, B, are predicted human and object boxes, respectively. The training objective is to opti- mize the parameters 0 by minimizing the expected HOI classification loss over matched queries:
f = arg min Er-x (B)
(2)
where by denotes the HOI detector, LM is the multi-label classification loss computed over the matched queries M, and X' is the training dataset, I represents the input image.
3.2
MODEL ARCHITECTURE.
Object Detector and Vision Encoder. To ensure architectural generality. we do not use the dataset- specific object queries produced by fine-tuned DETR. Instead, we obtain unary queries via ROI Align, using the feature maps extracted from VLM pretrained on large-scale data and the detected bounding boxes. Specifically, given human and object boxes B, and B,, we apply ROI Align on the feature map F I. IR=WXX to extract the corresponding unary queries Qh and Qo. both of dimensionality C.
Human-Object Geometrical Encoder. Since VLMs such CLIP are pre-trained with image- level contrastive objectives, they primarily capture global semantic information. Compared to the object queries in DETR, their spatial awareness and region-level discrimination are relatively lim- ited, which motivates the introduction of Geometry each bounding box using its normalized center and size with sinusoidal positional encoding. Specif- ically, given bounding box B = (æ1, y1, F2, U2) and image size (W, H), we normalize the box as 11,
query refinement. To this end, we encode
B, We then compute the center and size as C and S: B W W 2/ C (01 f22 0n +22), S (it- 21.92 01). Apply sinusoidal encoding to each: PE(B) = PE(C) @ PE(S), where PE(-) the standard 2D sinusoidal positional encoding function and t denotes vector concatena- tion. The resulting spatial embeddings are added to the unary queries Qn and Qo. and further refined through a self- attention mechanism. This process can be expressed as: Q = Self-Anm + PE(B)), Next, construct pairwise human-object queries Qi-o- Specifically, we iterate over all possible human-object combinations and concatenate their features to form paired queries, expressed as:
Q Liinear(C(Qu Q,l), where E H, EOUR H.
Here, H and O denote the sets of detected humans and objects, respectively. The notation C denotes concatenation. The inclusion of j E H allows the model to capture human-hume interactions.
Previously, defined unary positional encodings. To enrich the pairwise representation with instance-level spatial awareness, we further encode the geometric relation between each human- object pair using pairwise spatial encoding, following UPT (Zhang al., 2022a). This produces
4

---

Published as a conference paper at ICLR 2026
Student Medel
Pretrained
spane GT sqperebe
our dense LIVE supervise
From Scrttch
Teacher Model
captures the spadial and semantle driberite and fifermooos
LANE prosides dense saperviioo for all posiible 11-0
Object Detector
Essoder
Objezi Detector
Encialet
GT HOt antural langaege person peing basebuli but" boiding baneball
Selegee
XD oerr
Sporial
Licoder
H10 Geometrical Encoder
IIO Geometical
Prooder
0.37
0.33
cosler siadanity
Macced HID- Query Seleiioe
Isceneedd Centrasive Iives
XD
sorler clwdlarlyy
VLM Lisking
VLM Linking
Decoder
AD
Decoder
Figure 3: Progressive Learning Strategy (a) Student-teacher framework: the student model is trained from scratch with sparse GT supervision, while the pretrained teacher provides dense guid- ance via multi-level knowledge distillation. (b) Three levels of knowledge alignment: logits-level, query-level, and feature-map level, where instance-level matching enables fine-grate mne-grained supervision on spatial and semantic representations.
spatial relation vector Ri,, for each pair (i,j). We then fuse the queries Qh-o with their correspond- ing spatial encodings Rz, through a Multi-Modal Fusion (MMF) module. The fusion process is defined as: x LN (FC) y (Ng(FFC( and MLP (LL((ReUU(Conead/ vl)), where FC1, FC2 are linear projections to shared embedding space, LN,, I.2, are LayerNorm lay- ers, and MIPP is a multi-layer perceptron that produces the final fused representation. The output serves the final output of the encoder, i.e., the refined pairwise query Qh-o.
At this stage, we decouple the features from the object detector and derive queries solely based on VLM features and detected bounding boxes, thereby mitigating detector-inducet biases. Mean- while, it enhances spatial awareness and produces paired H-O instances for subsequent reasoning.
VLM Linking Decoder. In the decoder, we perform cross-attention between the pairwise queries Qi-o and the VL. feature map F E H W xC to aggregate both spatial cues and semantic infor- mation. Building standard cross-attention, we design VLM Linking structure consisting of spatial branch and a semantic branch.
The spatial branch reduces the dimensionality of the feature map through connector module, forming a latent bottleneck that interacts with Qi-o- This branch focuses on capturing fine-graine grained geometric relationships and spatial cues within the scene. To further enhance spatial reasoning, adopt the intentio-gguddnne mechanism proposed in PViC (Zhang et al., 2023), where positional encodings derived from bounding boxes are used constrain the attention maps.
In contrast, the semantic branch expands Qh and performs attention in the high-dimensional native space of the VLM, enabling richer aggregation of high-level semantics. While the spatial branch enhances fine-grained modeling, the semantic branch leverages global representations to improve transferability. The outputs of both branches are fused by concatenation followed by feed-forward network:
Ph- Lincar Linoar(Qn-o) F' = MLP(F).
(4)
Q MLP (CA' (Qn-o0 CA(QT.
(5)
where pl denotes the compressed feature map from the latent branch, ML. is a multi-layer per- ceptron used for final fusion, and C indicates concatenation. Here, CA(.) refers to the standard cross-attention, while Ceb)()) is a box-encoding- passed through an FFN initialized with CLIP text embeddings generate the final HOI logits.
modified cross-attention. Finally, Q is
3.3 PROGRESSIVE LEARNING STRATEGY
We propose unified architecture that establishes strong baseline for adapting foundation models to HOI detection. Building upon this foundation, we further introduce a knowledge learning strategy based teacher-student paradigm alleviate the challenge of sparse supervision.
5

---

Published as a conference paper at ICLR 2026
First, we leverage architecture to construct pure baseline by pre-training it on HOI data using only the original ground-truth annotations, which serves as the Teacher Model. Thanks to the ar- chitectural design, the teacher model transforms the frozen image-level representations from VLMs into learnable instance-level representations tailored for HOI tasks.
Second, we employ the pre-trained model to perform teacher-student transfer paradigm, illus- trated in Figure 3 Left. The Student Model is jointly supervised by both ground-truth annota- tions and guidance from the pre-trained teacher. Since the teacher and student share the same input and architecture, we achieve one-to-one aligned human-object instances. This alignment en- ables knowledge transfer across all candidate human-object pairs, rather than being restricted the limited subset defined by matched queries.
As shown in Figure 3, traditional supervision (black dashed arrow) covers only limited set of an- notated human-object instances. In contrast, our learning strategy provides adaptive supervision for all potential instances (orange dashed arrow), delivering richer and more comprehensive guidance. Moreover, it facilitates knowledge transfer across multiple levels, further enhancing the models ability to capture complex interaction patterns. We adopt knowledge transfer losses with Kullback- Leibler (KL) divergence: ICDxL (Jsus fi) KL (o (Ji/r) ({Ss//))) where Ssu and fe denote the student and teacher features or logits, (-) is the softmax function, and T is a temperature factor.
Feature Map. We perform knowledge transfer the feature map level by aligning the repre- sentations produced by the VLMs using a lightweight adapter. Let Fam, F E Cxxx* de- note the feature maps from the student and teacher models, respectively. To address poten- tial mismatches in spatial and channel dimensions, the students. Specifically, we first apply bilinear interpolation to adjust the spatial resolution: F/ Interpolate( Fi, size (H W )), followed by trilinear interpolation to align the channel dimension: F" = Interpolate F", size = (Cau, Hau, Wau)). Both feature maps are then flattened into shape HW C, and the loss is computed as: LDD = KCD F.").
align the teachers feature map to match
unified architecture, we design fully mirrored student-teace teacher paradigm in which both the HO Geometrical encoder and VLM Linking decoder share identical structures. This symmetry enables query-level knowledge transfer across all human-object pair queries throughout the entire model. Formally, the query-level transfer loss is defined as:
Query. Thanks
Cauery
Σ
CD(
stu 2.
1 La
Σ
KD
(d distu
(6)
where Lae and La denote the number of layers the encoder and decoder, respectively. 2 and d represent the sets of student and teacher queries at the b-th layer of the encoder. Similarly, O(
and olt) denote the student and teacher queries at the f-th decoder layer. The function ICP(.,) computes the average token-wise distillation loss between corresponding query sets.
Logits. We perform knowledge transfer at the logits level, which provides multi-label interaction guidance for each human-object pair. This also captures richer contextual information in the distribu- tion of logits over potential human- interactions. Additionally, we follow previous two-stage methods (Zhang et al., 2023) and combine the predicted logits with detection confidence scores. The
final score is computed as: V llog I+exp(- -- where denotes the confidence score of the paired bounding boxes and O is the predicted HOI logit. The distillation loss the logits level is then defined as: Glugit KD (F)(Fa)), (F.)).
Now, we extend the original training objective by applying knowledge transfer across all human- object pairs and at multiple levels G, in the following objective:
0 arg min Er~x (I, + d
Σ (D, 9Eg
where 4, is the teacher model and ICD, denotes the knowledge distillation loss at level g, which includes feature map, query, and logits-level supervision.

---

Published as a conference paper at ICLR 2026
Table 1: Zero-shot performance comparison under RF-UC, NF-UC, and UO settings. HM denotes the harmonic mean. The best result in each column is in bold, the second best is underlined. The symbol indicates results with CLIP- VIT-L as the VLM.
Method
RF-UC
NF-UC
UO
HM
Unseen
Seen
Full
HM
Unseen
Seen
Full
HM
Unseen
Seen
Full
F Sully--upervised methods
PVIC (Zhang et al.)
27.85
24.45
32.36
30.78
26.80
24.74
29.23
28.07
25.07
19.13
36.37
33,50
HOICLIP (Ning et al.)
29.40
25.53
34,85
32.99
27.22
26.39
28.10
27.75
21.28
16.20
30.99
28.53
LOGICHOI (Li al.)
29.79
25.97
34,93
33.17
27.34
26.84
27.86
27.95
20.68
15.67
30.42
28.23
GEN-VLKT (Liao et al)
25.91
21.36
32.91
30.56
24.19
25.05
23.38
23.71
15:42
10.51
28.92
25.63
Zero-shol oriented methods
CM et
30.63
27.63
34.35
33.01
31.76
32.41
31.13
31.39
CLIP4HOI (Mao et al.))
31.59
28.47
35.48
34.08
29.72
31.33
28.26
28 90
32,25
31,79
32.73
32,58
BCOMf (Wang al.)
31.45
35.04
33.74
32.43
33.12
31.76
32.03
CMMP (Lei et al.)
31.07
29.45
32.87
32.18
30.85
32.09
29.71
30.18
32.40
33.76
31.15
31.59
EZ-HOI (Lei et al.)
31.38
29.02
34.15
33.13
33.66
30.55
31.17
32,66
33.28
32,06
32.27
HOLa (Lei et al.)
32.69
30.61
35.08
34.19
33.35
35.25
31.64
32.36
34.65
36.45
33.02
33.59
33.42
32.25
34.68
34.19
34.07
33.72
34.42
34.25
33.73
34.05
33.41
33.66
LINK
39,40
38.51
40.33
39,97
35.14
34.63
35.67
35.43
37.92
38.24
37.61
37.91
Table 2: Zero-shot comparison under UV setting.
Table 3: Few-shot comparison on HICO-DET and V-COCO datasets.
Method
UV
HM
Unseen
Seen
Full
Fully-supervised methods
HICODET-Full
VCOCO-AP3"
PVIC (Zhang et al.)
23.84
19.58
30.48
28.95
38-
HOICLIP (Ning et al.)
27.69
24.30
32.19
31.09
60
LOGICHOI (1. al.)
34-
59
GEN-VLK (Liao al.)
24.76
20.96
30.23
28.74
Zero-shot oriented methods
al.)
CLIP4HOI (Mao et al.)
28.35
26.02
31.14
30.42
BCOMF (Wang et ul.)
26
CMMP (Lei et al.)
29.13
26.23
32.75
31.84
EZ-HOI el al.)
28.64
25.10
33.49
32.32
22
ACABBM
CHMP
HOLa (Lei et al.)
31.09
27.91
35.09
34.09
PYC
4
12
16
32
6.6
12
32
LINK
29.29
27.01
32.00
31,30
Number training examples class
LINK
31,92
27.22
38,36
36.88
EXPERIMENTS
Implementation Details. We trained scales of object detectors for fair comparison with prior work: DETR (Carion et al., 2020) with a ResNet-50 (He et al., 2016) backbone, and H-Deformable- (Liu al., 2021) backbone. Both detectors
DETR (Jia al., 2023) with Twin-Transformer-L.aa are pretrained on MS-COCO (Lin et al., 2014) and fine-tuned on the target datasets following stan- dard practice. ROI-aligned features are extracted
resolution of 7 x 7 using average pooling- We adopt A with a weight decay of 10-4, training for 15 epochs with FocalBCE loss. The learning rate decayed to 20% its initial value after the 10th epoch. For KDD losses, the temper- ature coefficient is set to 2.0. In our main comparisons, we follow prior work and use CLIP as the visionlanguage backbone. The connector module projects features into a 384-dimensional space. Unless otherwise specified, the teacher model is CLIP VTT-L/14@336px. Additional details are provided in the supplementary material.
4.1 COMPARISON WITH STATE-OF-THE-ARTY STATE-OF-THE-ARTS
Zero-shot Results. Table and 2 reports zero-shot performance under four standard settings: RF- UC (Rare-First Unseen Combination). NF-UC (Non-Rare First Unseen Combination), UO (Unseen Object) and UV (unseen Verb). HM denotes the harmonic mean between seen and unseen categories. Our method consistently outperforms both sully-sup and zero-shot-oriented approaches. In our experiments, LINK uses CLIP ViT-Base for both teacher and student, while LINK uses CLIP ViT-Large. Across the four zero-shot settings, LINK achieves two best and two second-best results among SOTA methods. Under the RF-UC setting, our method achieves an unseen score of 32.25, surpassing the previous best result of 30.61 61 by +1.64. When scaling up the ViT-Large, the perfor- mance further improves. Similarly, under the NF-UC and UO settings, our model attains harmonic means of 35.14 and 37.92, respectively, establishing state-of-the-ar performance. These results

---

Published as a conference paper at ICLR 2026
Table 4: Comparison with state-of-the-art on HICO-DET and v-COCo. Results are grouped into fully-supervised v.s zero-shot oriented methods. Bold indicates the best performance. All experiments are conducted under a fair setting where the teacher and student use the same scale VLM, such as CLIP-ViT-B to CLIP-ViT-B.
Method
Configuration Backhone VLM
HICO-DET (Default)
(Known Obj.) Rare
V-COCO
Full
Rare
N-rare
Full
N-rare
AP
Fully-supervised methods
KI2HOI (Xue et al.)
R50/ CLIP-B
34.20
32.26
36.10
37.85
35.89
38.78
65.0
HOICLIP (Ning
al.)
R50/ CI.IIBB
34.69
31.12
35.74
37.61
34.47
38.54
64.8
CLIP4 (Mao et al.)
R50/ CLIP-B
35.33
35.74
35.27
37.77
66.3
LOGICHOI (Li al.)
R50/ CLIP-B
35.47
32.03
36.22
38.21
35
39.03
65.6
DP-ADN (Gao el.)
RS0/ CLIP-B
35.91
35.82
35.94
38.99
39.61
38.80
64.8
HORP (Geng et al.)
CLIPL
38.61
36.14
39.34
40.98
38.25
41.79
68.3
InterProDa (Jia al.)
R50 CLIP-L
42.67
45.21
41.92
De (Yang el al.)
/
42.93
42.41
43.11
44.20
45.23
72.1
PVIC (Zhang et al.)
Swin-L/-
44.32
44.61
44.24
47.81
48.38
47.64
68.0
MP-HO1 (Yang al.)
Swin-L/CLIP-LAn
44.53
44.48
44.55
HORP (Geng et al.)
Swin-L/CLIIP-1
47.53
46.81
47.74
51.24
50.78
$1.38
71.1
Zero-shot oriented methods
CMMP (Lei al.)
R50/ CLIP-B
33.24
32,26
33.53
36.32
34.87
36.75
61.2
ADA-CM (Lei al.)
R50/ CLIP-B
33.80
31.72
34.42
37.06
35.43
37.55
61.5
EZ-HOI (Lei al.))
R50/ CLIP-B
33.15
29.11
34.36
36.38
31.93
37.71
63.5
HOLa (L.ei et al.)
R50/ CLIP-B
35.41
34.35
35.73
38.59
36.43
39.10
LAIN (Kim al.)
/ CLIP-B
36.11
39.47
LINK
R50/ CLIP-B
37.43
37.18
37.50
40.46
40.30
40.51
66.5
CMcP
R507 CLI/LL
38.14
38.25
41.16
ADA-CM (Lei at.)
R50/ CLIP-L
38.40
37.52
38.66
41.25
40.41
41.50
64.0
EZ-I1OI (Lei al.)
R50/ CLIP-L
38.61
37.70
38.89
41.65
40.75
41.91
65.4
HOLa (Lei et al.)
250/CLIP.L
39.05
38.66
39.17
42.13
41.18
42.42
66.0
LINK
//50/CLPP
42.92
45.03
42.20
45.79
47.00
45.67
68.1
Uni
40.11
42.08
BC-HOI (Hu et al.)
R50 20-PT----
43.01
45.76
42.18
45.35
47.94
44.57
70.6
LINK
R50 BLIP-2-OPT-2.70
43.72
45.82
43.10
46.11
47.71
45.62
68.5
CMMP (Lei al.)
Swin
44.26
43.48
43.89
47.15
48.36
46.79
63.3
ADA-CM (Lei al.)
Swin-L/CLIP
44,99
45.98
44.69
47.77
49.08
47.38
65.7
EZ-HOI (Lei et al.)
Swin-L/ CLIPL
45.22
46.15
44.94
47.63
47.51
66.1
HOLa (Lei et al.)
Swin-L/ CLIPPL
36.17
34.39
36.70
38.48
36.32
39.13
LINK
Swin-L/ CCLIPLL
49.06
53.63
47.60
51.34
56.29
49.86
69.2
Table 5: Comparison SWiG-HOI, demonstrating the open-vocabulary capability.
Method
N-rare
Rare
Novel
Full
Wang (Wang al., 202
10.93
6.63
2.64
7.98
THID (Wang et al., 2022)
17.67
12.82
10.04
13.26
AMP-HOL (Xue et al., 2024a)
19.77
14.00
9.74
14.29
MP-HOI-S (Yang
2024)
20.28
14.78
12.61
GEN-VLKT KT (Liao et al., 2022)
20.91
10.41
10.87
CMD-SE (Lei et al., 2024b)
21.46
14.64
10.70
15.26
SGC-Net (Lin et al., 2025)
23.67
16.55
12.46
17.20
LINK (ours)
24.37
17.88
12.15
17.97
highlight the strong generalization ability of our method to unseen HOI categories and validate its robustness in open-world scenarios.
Few-shot Results. We further evaluate few-shot performance HICO-DET and V-COCO against
oofte--ofthhharrr methods, as illustrated in Figure 3. Our method consistently achieving the best re- sults across both benchmarks from 1-shot to 32-shot settings. Interestingly, we observe trade-off' in prior works: fully- sullevsupe method PVIC performs well on V-COCO but lags on HICO-DES whereas pero-shot-orienter method ADA-CM shows the reverse trend. We attribute this inconsis- tency to dataset scale differences that V-COCO contains only 24 HOI categories, while HICO-DET comprises 600. In contrast, our model maintains strong performance across both datasets.
Comparison under Fully-Supervised Settings. Main results on HICO-DET and V-COCO Benchmarks are presented in Table 4. On HICO-DET, our method achieves state-of-the-art perfor- mance across different model scales, including R50+ViT-B. R50+ViT-L, and Swin-L+ViT-LL Our small-scale model achieves 37.43 and 37.18 mAP the Full and Rare subsets, respectively, out- performing the previous best query-free method (33,80/ 31.72 mAP) with relative improvements of 10.7% and 17.2%. Notably, while retaining the flexibility benefits of query-free designs, our method
8

---

Published as a conference paper at ICLR 2026
Full HOIs
Long-tail HOIs (less than 10 samples)
CLIP
BINO
Sigt
882 + LINK(OU))
BLIP
DINOg
Florence2
44
414
$1.7
12
1000
40
16.
34
11.
CUP
aueg
BUN
nIsozza
18
Satrr3
aigh124
Flarencs2-0
Morescs24
cure
cuP
aues
BUN
nINosza
DISoR4 18-8
saurra
aigur24
Flrreneelll
Horemoz24
DISD
Figure Experiments on Diverse Foundation Models. Our method (+LINK, striped bars)
consistently improves the performance of all baselines with various foundation models: includ- ing contrastive vision-language learning- Cernn and BLIP: self-supervised learning- -IN0v22 and DINO @ 448; and multitask multimodal pretraining-SigLPP and Florence2. Notably, LINK pro- vides the largest gains on long-tail HOIs (5 10 samples), highlighting its generalization capability.
Table 6: Ablation on HICO-DET under fully- Table 7: Ablation under zero-shot RF-UC set- supervised setting.
ting.
Encoder
Desoder
Full
N-Rare
Encoder
Decoder
HM
Unseen
Seen
Full
A1 Self-Attm
Cross-Amu
36.10
33.67
36.97
Self-Atth
Cross-Attn
31.74
30.23
33.36
32.54
Sell--Atmn
VLM-Link
39.23
39.02
Self-Attn
VLM--Lnk
35,47
34.78
36.17
35.84
A3
Goometrical
Cross-Attn
38.30
35.46
39.31
A3
Geometrical
Cross-Attn
34.55
33.54
35.60
35.10
Geometrical
VLM-Link
41.20
41.43
41.13
A4
Geometrical
VLM-Link
36.89
36.21
37.60
37.32
A5
LLogi-tevel KD
41.89
43.82
41.27
QQucry-lveel KD
42.34
43.62
41.84
+ Logit-level
37.77
37.24
38.31
38.10
A7
Map-level KD
42.92
45.03
42.20
Query-level KD
39.10
38.30
39.92
39.48
A8
multi-teacher (CLIP+ SigLIP)
43.54
45.58
42.93
A7
Map-level KD
39.40
38.51
40.33
39.97
surpasses even all specific-query based methods on standard benchmarks. When scaling up to Swin- L+VIT-L, our model achieves 49.06 mAP, a +1.53 mAP gain over the prior best (HORP 47.53), highlighting excellent scalability, On V-COCO, our method also achieves competitive performance and outperforms prior two-stage methods, including ADA-CM (Lei et al., 2023) and PViC (Zhang et al., 2023) by a large margin.
Comparison on vonn-Vocabulary benchmark SWIG-HO1. As shown in Table 5, our method achieves a new SOTA with 17.97 mAP on the full set, outperforming the previous best by 2.71 mAP (a relative 17.8%). We also obtain the best results on both Non-Rare (24.37 mAP, +13.6%) and Rare (17.88 mAP, +22.1%) subsets. Notably, our method achieves 12.15 mAP on the novel HOIs. demonstrating strong generalization capability.
In summary, the above comparative experiments demonstrate that our method excels in both spe- cialization (three standard benchmarks) and generalization (zero-shot and few-shot settings), without sacrificing one for the other.
Ablation Studies. We conduct ablation studies on HICO-DET under both fully-supervised and zero-shot RF-UC settings, as shown in Table 6 and Table 7. Notably, Our bascline (A1) adopts standard self-attention encoder over ROI features and a cross-attention decoder that attends to VLM representations, forming plain baseline. First, we evaluate the impact of our architectural design. Introducing the HO Geometrical Encoder (A3) or VLM Linking decoder (A2) each improves per- formance over the baseline (AI), while combining both (A4) yields the best results, confirming their complementary benefits. Second, we investigate our instance-level knowledge learning strat- egy. Progressive integration of logit-level (A55,, query-Ievel (A6), and map-level (A7) distillation further boosts performance. Finally, employing multiple teachers (A8) achieves the highest gain in the fully-supervised setting, while in the zero-shot setting, instance-level learning still provides improvement, raising the harmonic mean by +2.51. These results demonstrate the effectiveness of both our architectural components and the proposed knowledge learning strategy.
Various Foundation Models. Beyond CLIP, we further evaluate our method across diverse set of foundation models with different pre- pre-training paradigms, including contrastive learning (BLIP), vision-only self-supervised learning (DINOv2, DINO@ 448), and multitask multimodal pre-training

---

Published as a conference paper at ICLR 2026
(SigLIP2, Florence2). As illustrated in Figure 4, method successfully adapts to all these models within a unified architecture, establishing strong baselines (solid bars different colors). Further- more, by incorporating our proposed learning strategy, performance is consistently and significantly improved (striped bars). Notably, +LINK enhances the detection of rare interactions, an essential capability for real-world HOI applications where long-tail categories common and critical.
Table 8: Comparison of various OVD models on HICO-DET (Default and Known-Object settings).
Open-vocabulary
HICO-DET (Default)
HICO-DET (KO)
Scale
Object Detectors
Full
Rare
N-rare
Full
Rare
N-rare
YOLOvs-wordvv2
Small
27.07
29.83
26.24
30.06
32.37
29.37
wor
Medium
30.35
33.40
29.45
33.39
36.00
32.62
Large
32.64
36.51
31.49
35.98
40.06
34.76
YOL w0..0---wordv
Xlarge
33.73
37.30
32.66
36.98
40.65
35.88
Grounding-DIe DINO
Swin-tiny
32.97
38,21
31.40
35.96
40.76
34,52
Grounding- DINO
Swin- Swin-base base
39.69
45.99
37.81
42.42
47.94
40.77
Qwen3-VI,
8B
39.61
37.99
40.42
Plug-and-Play with OVDs. LINK operates detector-agnostic plug-and-play module over bitrary open-vocabulary detectors (OVDs), requiring no retraining or adaptation of the underlying detector. Given bounding boxes from any OVD, LINK performs HOI reasoning independently of detector-specific representations. As shown in Table 8, integrating LINK with off-the-shelf OVDs, including YOLOv8- Worldy2, YOLOvv8-Woldyy2, Grounding-DINO, and Qwen3-VI, yields strong and consistent per- formance across model scales.
CONCLUSION
In this paper, we presented a unified HOI detection framework that combines HO Geometrical En- coder with a VLM Linking decoder, enabling seamless integration with diverse foundation models and object detectors while ensuring consistent performance across different scenarios. To address the limitations of sparse annotations, we introduced an instance-level knowledge learning strategy under a teacher-student paradigm, which provides dense and adaptive supervision across all hu- manobject pairs. Extensive experiments on HICO-DET, V-COCO, and SWiG-HOI demonstrate that our method achieves state-of-the-ar performance and generalizes well across settings. Importantly, +LINK substantially improves the recognition of rare interactions, highlighting its value for real- world HOI applications where long-tail categories are both common and critical.
ACKNOWLEDGEMENT
This work is supported by the research fund under Grant No. 20242910035 from the Tsinghua Jniversity-Jiangsu CRRC Digital Technology Co., Ltd. Joint Research Center for Data Driven Intelligence of Industry.
REFERENCES
Yichao Cao, Qingfei Tang, Xiu Su, Song Chen, Shan You, Xiaobo Lu, and Chang Xu. Detecting any human-object interaction relationship: Universal hoi detector with spatial prompt learning on foundation models. Advances in Neural Information Processing Systems, 36:739-751, 2023.
Yichao Cao, Qingfei Tang, Xiu Su, Song Chen, Shan You, Xiaobo Lu, and Chang Xu. Detecting any human-object interaction relationship: Universal hoi detector with spatial prompt learning foundation models. Advances in Neural Information Processing Systems, 36, 2024.
Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In European conference on computer vision, pp. 213-229 Springer, 2020.
Yu-Wei Chao, Yunfan Liu. Xieyang Liu, Huayi Zeng, and Jia Deng. Learning to detect human- object interactions. In 2018 ieee winter conference on applications of computer vision (wacr), pp. 381-389. IEFE, 2018
10

---

Published as a conference paper at ICLR 2026
Junwen Chen and Keiji Yanai. Qahoi: query-baser based anchors for human-objec interaction detection. arXiv preprint arXiv:2112.086/44 2021.
Mingfei Chen, Yue Liao, Si Liu, Zhiyuan Chen, Fei Wang, and Chen Qian. Reformulating hoi detection as adaptive set prediction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9004-9013, 2021.
Chen Gao, Yuliang Zou, and Jia-Bin Huang. ican: Instance-centric attention network for human- object interaction detection. arXiv preprint arXiv: /808. 188888803377 /0437, 2018.
Chen Gao. Jiarui Xu, Yuliang Zou, and Hun-Bin Huang. Drg: Dual relation graph for human-object
interaction detection. In Computer Fision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XII 16, pp. 696-712. Springer, 2020.
Jiayi Gao, Kongming Liang, Tao Wei, Wei Chen, Zhanyu Ma, and Jun Guo. Dual-prior augmented decoding network for long tail distribution in hoi detection. In Proceedings of the AAAI Confer- ence on Artificial Intelligence, volume 38, pp. 1806-1814, 2024.
Pei Geng, Jian Yang. and Shanshan Zhang. Horp: Human-object relation priors guided hoi detection. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 25325 25335, 2025.
Georgia Gkioxari, Ross Girshick, Piotr Dollár, and Kaiming He. Detecting and recognizing human- object interactions. In Proceedings of the IEEE conference on computer vision and pattern recog- nition, pp. 8359-8367, 2018.
Tanmay Gupta, Alexander Schwing, and Derek Hoiem. No-frills human-object interaction detec- tion: Factorization, layout encodings, and training techniques. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 9677-9685, 2019.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 770-778, 2016.
Jiun Tian Hoe, Xudong Jiang, Chee Seng Chan, Yap-Peng Tan, and Weipeng Hu. Interactdiffusion: Interaction control in text-to-image diffusion models. In Proceedings of the IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition, pp.
2024.
Yupeng Hu, Changxing Ding, Chang Sun, Shaoli Huang, and Xiangmin Xu. Bilateral collaboration with large vision- languale models for open vocabulary human-object interaction detection. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 20126-2011 201666. 2025.
Ding Jia, Yuhui Yuan, Haodi He, Xiaopei Wu, Haojun Yu, Weihong Lin, Lei Sun, Chao Zhang, and Han Hu. Detrs with hybrid matching. In Proceedings of the IEÉE/CVF conference on computer vision and pattern recognition, pp. 19702-19712, 2023.
Mingda Jia, Liming Zhao, Ge Li, and Yun Zheng. Orchestrating the symphony of prompt distribu- tion learning for human-object interaction detection. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pp. 3940-3948, 2025.
Bumsoo Kim, Taeho Choi, Jaewoo Kang, and Hyunwoo J Kim. Uniondet: Union-level detector towards real-time human-object interaction detection. In Computer Vision-ECCC 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XV 16, pp. 498- 514. Springer, 2020a.
Bumsoo Kim, Junhyun Lee, Jaewoo Kang, Eun-Sol Kim, and Hyunwoo J Kim. Hotr: End-to-end duman-objec interaction detection with transformers. In Proceedings of the IEEE/CVF confer- ence on computer vision and pattern recognition, pp. 73- 2021.
Dong-Jin Kim, Xiao Sun, Jinsoo Choi, Stephen Lin, and In So Kweon. Detecting human-object interactions with action co-occurrence priors. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XXI 16, pp. 718-736. Springer, 2020b.
11

---

Published as a conference paper at ICLR 2026
Sanghyun Kim, Deunsol Jung, and Minsu Cho. Locality- aware zero-shot human-object interac- tion detection. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 20190-20200, 2025.
Qinqian Lei, Bo Wang, and Tan Robby T. Ez-hoi: Vim adaptation via guided prompt learning for zero-shot hoi detection. In The eighth Annual Conference on Neural Information Process- ing Systems, 2024a.
Qingian Lei, Bo Wang, and Tan Robby T. Hola: Zero-shot hoi detection with low-rank decomposed vlm feature adaptation. In In Proceedings of the IEEE/CVF international conference computer vision, 2025a.
Ting Lei, Fabian Caba, Qingchao Chen, Hailin Jin, Yuxin Peng, and Yang Liu. Efficient adap- tive human-object interaction detection with concept-guided memory. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 6480 6490, 2023.
Ting Lei, Shaofeng Yin, and Yang Liu. Exploring the potential of large foundation models for open- vocabulary hoi detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 16657-16667, 2024b.
Ting Lei, Shaofeng Yin. Yuxin Peng, and Yang Liu. Exploring conditional multi-modal prompts for zero-shot hoi detection. In European Conference on Computer Vision, pp. 1-19. Springer, 2025b.
Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language-image pre- training for unified vision-language understanding and generation. In International conference on machine learning, pp. 12888- 12900. PMLR, 2022.
Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In International conference on machine learming, pp. 19730-19742. PMLR, 2023a.
Liulei Li, Jianan Wei, Wenguan Wang, and Yi Yang. Neural-logic human-object interaction detec- tion. Advances in Neural Information Processing Systems, 36:21 11--21588-2111 171, 2023b.
Yue Liao, Si Liu, Fei Wang, Yanjie Chen, Chen Qian, and Jiashi Feng. Ppdm: Parallel point de- tection and matching for real-time human-object interaction detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 482-490, 2020.
Yue Liao, Aixi Zhang, Miao Lu, Yongliang Wang. Xiaobo Li, and Si Liu. Gen-vlkt: Simplify asso- ciation and enhance interaction understanding for hoi detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 20123-20132, 2022.
Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects context. In Computer Vision-ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13, pp. 740-755. Springer, 2014.
Xin Lin, Chong Shi, Zuopeng Yang, Haojin Tang, and Zhili Zhou. Sge-net: stratified granular comparison network for open-vocabulary hoi detection. In Proceedings of the Computer Vision and Puttern Recognition Conference, pp. 4539-4549, 2025.
Wen Liu, Weixin Luo, Dongze Lian, and Shenghua Gao. Future frame prediction for anomaly detection-a new baseline. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 6536-6545, 2018.
Yang Liu, Qingchao Chen, and Andrew Zisserman. Amplifying key cues for human-object- interaction detection. In European Conference on Computer Vision, pp. 248-265. Springer, 2020.
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hicrarchical vision transformer using shifted windows. In Proceedings of the IFFF/CVF international conference computer vision, pp. 10012-10022, 2021.
12

---

Published as a conference paper at ICLR 2026
Yunyao Mao, Jiajun Deng, Wengang Zhou, Li Li, Yao Fang, and Houqiang Li. Clip4hoi: towards adapting clip for practical zero-shot hoi detection. Advances in Neural Information Processing Systems, 36:45895-444 445966, 2023.
Shan Ning, Longtian Qiu, Yongfei Liu, and Xuming He. Hoiclip: Efficient knowledge transfer for hoi detection with vision-language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 23507-23517, 2023.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp. 8748 8763. PMLR, 2021.
Guangzhi Wang, Yangyang Guo, Ziwei Xu, and Mohan Kankanhalli. Bilateral adaptation for human-object interaction detection with rcclusion-robustness. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 27970-27998 2024.
Suchen Wang, Kim-Hui Yap, Henghui Ding, Jiyan Wu, Junsong Yuan, and Yap-Peng Tan. Discov- ering human interactions with large-vocabulary objects via query and multi-scale detection. In Proceedings of the IEEE/CVE International Conference on Computer Vision, pp. 13475-13484, 2021a.
Suchen Wang, Yueqi Duan, Henghui Ding, Yap-Peng Tan, Kim-Hui Yap, and Junsong Yuan. Learn- ing transferable human-object interaction detector with natural language supervision. In Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 939-948, 2022.
Zirui Wang, Jiahui Yu, Adams Wei Yu, Zihang Dai, Yulia Tsvetkov, and Yuan Cao. Simvlm: Sim- ple visual language model pretraining with weak supervision. arXiv preprint arXiv:2108. 10904, 2021b.
Eastman ZY Wu, Yali Li, Yuan Wang, and Shengjin Wang. Exploring pose-aware human-object interaction via hybrid learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 17815-17825, 2024.
Mingrui Wu, Jiaxin Gu, Yunhang Shen, Mingbao Lin, Chao Chen, and Xiaoshuai Sun. End-to-end zero-shot hoi detection via vision and language knowledge distillation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pp. 2839-2846, 2023.
Kejun Xue, Yongbin Gao, Zhijun Fang. Xiaoyan Jiang, Wenjun Yu, Mingxuan Chen, and Chen- mou Wu. Adaptive multimodal prompt for human-object interaction with local feature enhanced transformer. Applied Intelligence, pp. 1-13, 2024a.
Weiying Xue, Qi Liu, Qiwei Xiong, Yuxiao Wang, Zhenao Wei, Xiaofen Xing, and Xiangmin Xu. Towards zero-shov human-object interaction detection via vision-language integration. arXiv preprint ar 4003v:2403.077 077266, 2024b.
Bin Yang, Yulin Zhang, Hong- Yu Zhou, and Sibei Yang. No more sibling rivalry: Debiasing human- object interaction detection. arXiv preprint 000.0:22000000000 2025.
Jie Yang, Bingliang Li, Ailing Zeng, Lei Zhang, and Ruimao Zhang. Open-world human-object interaction detection via multi-modal prompts. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 16954-16964, June 2024.
Frederic Z Zhang, Dylan Campbell, and Stephen Gould. Efficient two-stage detection of human- object interactions with novel unary-pairwiss transformer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 20104-20113 12, 2022a.
Frederic Z Zhang, Yuhui Yuan, Dylan Campbell, Zhuoyao Zhong, and Stephen Gould. Explor- ing predicate visual context in detecting of human-object interactions. In Proceedings of the IEEF/CV International Conference Computer Vision, pp. 10411-, 2023.
13

---

Published as a conference paper at ICLR 2026
Haotian Zhang, Pengchuan Zhang, Xiaowei Hu, Yen-Chun Chen, Liunian Li, Xiyang Dai, Lijuan Wang, Lu Yuan, lenq-Neng Hwang, and Jianfeng Gao. Glipv2: Unifying localization and vision- language understanding. Advances in Neural Information Processing Systems, 35:36067-36080, 2022b.
Penghao Zhou and Mingmin Chi. Relation parsing neural network for human-object interaction detection. Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 843 843-851,200 2019.
Cheng Zou, Bohan Wang, Yue Hu, Junqi Liu, Qian Wu, Yu Zhao, Boxun Li, Chenguang Zhang, Chi Zhang, Yichen Wei, et End-to-end human object interaction detection with hoi transformer. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp- 11825- 1834, 2021.
14

---

Published as a conference paper at ICLR 2026
A APPENDIX
A.1 OVERVIEW OF THE PROPOSED LINK
To develop a VLM-based HOI detector that performs robustly across both specific and generalizable scenarios, our work is guided by two central questions: How can we design unified and universal HOI detection architecture that maintains consistent performance across different settings? How can we generate dense and informative supervision signals to better guide HOI learning, especially under sparse annotations? ? To this end, we propose the Learning Instance-level Knowledge frame- work (LINK), a generalizable and modular architecture for HOI detection built upon vision-language models (VLMs). As illustrated in Fig. 2 of the main paper, LINK consists of two key components: a HO Geometrical Encoder that models pairwise spatial relationships between humans and ob- jects to enhance contextual reasoning, and a VLM-Linking Decoder that fuses native and latent representations for robust HOI prediction across diverse tasks. On top of this architecture, we intro- duce an Instance-level Knowledge Learning strategy that adopts a fully mirrored teacher-student paradigm. This design enables multi-level supervisionspaniing features, queries, and logitsover all candidate human-object pairs. Unlike traditional methods that only supervise ground-truth matches, our strategy generates adaptive and learnable signals to distinguish both positive and negative in- stances at fine granularity. Together, this unified design ensures plug-and-pla compatibility with different object detectors and foundation models, while our dense supervision paradigm enhances generalization to rare, zero-shot, and open- vocabulaay settings. The proposed LINK framework thus provides a scalable, transferable, and high- perforaefo solution for modern HOI detection.
A.2 DATASETS FOR HOI DETECTION
HICO-DET
V-COCO
SWIG-110
Jamplax
Propurecy CON POAC
130
20
Figure 5: Distribution of widely-used HOI datasets: HICO-DET, V-COCO and SWiG-HOL.
Our experiments are conducted on three widely used HOI detection benchmarks: HICO-DET, V- COCO, and SWIG-HOL. Below, provide detailed overview of each dataset in terms of compo- sition and scale.
HICO-DET is HOI detection benchmark extended from the HICO classification dataset. It in- cludes 600 HOI categories, formed from 80 object categories and 117 verb categories. The dataset contains total of 47,100 images (37.6k for training and 9.5k for testing), with approximately 151.2k annotated interaction instances (117.8k for training and 33.4k for testing). It supports diverse inter- action modeling and exhibits clear long-tail distribution 39 of the 600 HOI categories have fewer than 10 training samples. Evaluation is typically conducted under three settings: Full (all 600 cate- gories), Rare (long-tail HOIs), and Non- Rare (the remaining categories).
V-COCO is smaller-scale HOI dataset built on MS-COCO Lin et al. (2014). focusing on recogniz- ing common actions. defines 24 HOI categories involving 80 object categories and 24 verbs. The dataset consists of 10.3k images (5.4k for training and 4.9k for testing) and approximately 26.2k annotated interactions (13.8k for training and 12.4k for testing).
SWIG-HOT is a large-vocabulary HOI benchmark derived from the SWIG and DOH datasets. It features a highly diverse interaction space and vocn-vaay combinations, making it suitable for evaluating generalization and long-tail performance. It defines 14.130 HOI categories from 1,000 object categories and 407 verb categories. The dataset includes 54.6k training images and 13.6k test images, with total of 99.8k annotated interaction instances (80.2k for training and 19.6k for testing).
Overall, HICO-DET serves widely adopted benchmark; V-COCO offers compact annotations in standardized COCO scenes; and SWiG-HOI is designed for large-scale open-world evaluation. Our method, with its unified architecture, is capable of adapting to diverse scenariosincluding open-
15

---

Published as a conference paper at ICLR 2026
Ours
PViC (Zhang et al., 2023)
Detector
Full
Rare
N-rare
Full
Rare
N-rare
DETR-R50*
43.54
45.58
42.93
34.32
31.62
35.13
Deformable-DEEe Deribll---ERR-RSS -50
41.66
45.65
40.46
18.72
13.49
20.28
H-Deformable- Derell--EER-iiiii
49.06
53.63
47.60
15.83
10.94
17.29
Y1LO,12-nam
27.12
25.77
27.52
14.42
10.72
15,51
YOLO
30.62
28.46
31.13
13.63
9.90
14.75
Table 9: Cross-detector evaluation HICO-DET. Both methods trained with DETR-R50 (+). Our method generalizes well to new detectors without fine-tuning, while PVIC degrades signif- icantly, showing its reliance on detector-specinfe features.
vocabulary, zero-shot, few-shot, and low-label settings. Its effectiveness has been thoroughly evalu- ated across all three datasets.
A.3 ADAPTABILITY TO OBJECT DETECTORS
Thanks to our architecture's fully decoupled design, LINK is inherently adaptable to a wide range of object detectors. Specifically, model does not rely any detector-specific features inter- mediate representations. Instead, it solely requires bounding boxes as input and performs spatial encoding based on the geometry of human-object pairs.
This design enables plug-and-play compatibility with arbitrary object detectors, such as Faster R-CNN, YOLO, DETR, Deformable DETR even visual grounding MLLMs, without requiring any fine-tuning or feature alignment. As a result, LINK provides flexible and efficient solution that can be readily deployed across different detection backbones and application domains, while maintaining consistent performance and minimizing adaptation overhead.
Table presents a cross- detssdet evaluation on the HICO-DET dataset to assess the adaptability of our architecture. The results compare our method (top) and PViC Zhang et al. (2023) (bottom) under five different object detectors, including DETR-R50*, Deformable DETR, H-Deformable DETR with Swin-L backbone, and two YOLO variants.
The rows marked with * indicate that both methods are trained using DETR-R50 as the object detector. When directly replacing the detector inference timewithout any additional fine-tuningour method maintains robust performance across all detectors. This highlights the plug-and-play nature of our architecture, which is fully decoupled from detector-specife features and relies solely bounding box inputs for interaction reasoning. In contrast, PViC and similar prior methods exhibit significant performance degradation when the detector is changed, with mAP dropping to nearly unusable levels. This illustrates their strong coupling with detector-specific features and the need for costly re-training whenever the detection backbone modified.
These results demonstrate that our method offers superior flexibility and generalization, making it more practical for real-world deployment.
A.4 MORE IMPLEMENTATION DETAILS
Training Setup. We build our implementation upon the official open-source codebase of PViC Zhang et al. (2023), extending to support our proposed LINK framework. All experiments are conducted using PyTorc with 8 NVIDIA RTX 4090 GPUs (24GB each). Our method is hardware- friendly: we train the model for 15 epochs with a total batch size of 16, using 4 GPUs under Distributed Data Parallel (DDP) training. Following PViC, we adopt the same preprocessing and loading pipeline for the HICO-DET and V-COCO datasets. Additionally, we generalize the data interface and configuration system support arbitrary datasets in the same manner, including the large-scale SWIG-HOI dataset introduced earlier.
Hyper-parameters. We adopt the Focal Loss for classification with two key parameters: alpha 0. 5, gamma 0. 1. controlling the loss weighting and focusing factor. We filter pre- dicted instances using confidence threshold of box-score-thresh 05, and retain
16

---

Published as a conference paper at ICLR 2026
dynamic number of predictions per image within the range of min-instances max-instances 15. the zero-shot setting, we apply a top-k filtering strategy for can- didate verbs and objects, with zs-topk 10 and a scaling factor 28-topk-factor to dynamically adjust the numher of retained interaction candidates based on instance confidence.
to
1.8
Loss Details. As described in the main paper, training objective combines standard focal loss for HOI classification with a set of auxiliary losses designed to support instance-level knowledge learning under teacher-student paradigm. The overall objective given by:
arg min E | (L, + Σ () gEg
Here, L. denotes the focal loss applied to the main classification outputs, and KD, represents set of knowledge distillation objectives at different levels E g, including logits, queries, and feature maps. Specifically, assign the following weights to the instance-level auxiliary losses:
loss_logite 1.0
(Ground Truth classification logits)
poss.query.nativee 1.0
(decoder native branch)
Loss.query.latent= 1.0
(decoder latent branch)
loss_feat_mapi 0.5
(intermediate feature maps)
Loss.query.encoder: 0.5
(encoder HO queries)
In practice, we find that varying these weights does significantly affect the model's final perfor- mance, indicating that our learning strategy is robust to hyperparameter settings. The chosen values are primarily aimed at balancing the relative magnitudes of each loss term, ensuring that no single component dominates the overall training objective.
These components collectively provide multi-level, instance-awen aware supervision across both encoder and decoder, helping the student model to more effectively distinguish ine-grained human-object interactions.
A.S COMPREHENSIVE COMPARISON WITH MULTIMODAL LARGE LANGUAGE MODELS
To enable fair and reproducible comparison, define standardized evaluation protocol and output format for applying Multimodal Large Language Models (MLLMs) to the HOI detection task. Since MLLMs typically produce textual outputs, we explicitly define both the input-output structure and formatting rules for extracting HOI predictions.
Output Format Definition. The MLLM is expected to produce a structured output in the following format:
HOI result
list of [xl, yl, x2, y2] "classes": [...]. # class IDs aligned with boxes list of hoi triplets ""human_idx"; int, "object_idx"; int, "verb": int),
"boxes": [...],
interactions"
Formatting Rules:
Indices human ix and object..idd ixx refer entries in the boxes list.
Class ID 0 corresponds human; others indicate object categories.
For multiple verbs, use multiple entries with shared indices and distinct verb IDs.
The output must be JSON-compatble.
We observe that directly prompting MLLMs to perform object detection and HOI predictioneither through supervised fine-tuning in zero-shot settingremains highly challenging due to their lim- ited spatial grounding capabilities. To address this, we provide MLLMs with the object detectors predictions (i.e., bounding boxes and class labels) part of the input prompt. This strategy enables
17

---

Published as a conference paper at ICLR 2026
Method
mF1
mPrec
mRec
Full
Rare
N.Rare
Full
Rare
N.Rare
Full
Rare
N.Rare
HOI
Detection Based on R50-DETR
OB Detection Prompting
Owen-vl-max
0.1719
0.1523
0.1777
0.2030
0.1585
0.2164
0.1833
0.1546
0.1919
Claude-3.S-sonne
0.1895
0.2429
0.1736
0.2071
0.2553
0.1927
0.2099
0.2554
0.1963
GPT-o4-mini
0.2082
0.2058
0.1958
0.2271
0.2120
0.2173
0.2419
0.2186
0.2337
GPT-o3
0.2758
0.3310
0.2592
0.2792
0.3507
0.2579
0.3328
0.3430
0.3297
Gemini-2.5-fae
0.2680
0.3347
0.2481
0.2754
0,3412
0.2557
0.3305
0.3659
0.3199
Gemini-2.5-pro
0.3161
0.4010
0.2908
0.3084
0.4145
0.2767
0.3937
0.4287
0.3833
CMMP (0.48B)
0.3324
0.3024
0.3414
0.3324
0.3024
0.3414
0.4352
0.3448
0.4622
ADA-CM (0.44B)
0.3414
0.3133
0.3498
0.3225
0.3181
0.3238
0,4469
0.3611
0.4715
Ours (0.52B)
0.3671
0.3542
0.3710
0.3351
0.3538
0.3296
0.4982
0.4149
0.5231
HOI Detection Based on Ground Truth OB Detection Prompting
Qwen2.5-v1-72b
0.2338
0.2486
0.2294
0.2801
0.2681
0.2836
0.2317
0.2440
0.2281
Owen-vl-max
0.2930
0.2605
0.3027
0.3320
0.2754
0.3489
0.2935
0.2606
0.3033
Claude-3.5-haiku
0.1000
0.0871
0.1038
0.1156
0.1001
0.1202
0.1302
0.1025
0.1384
Clande-3.5-sonnet
0.2652
0.3145
0.2505
0.2815
0.3333
0.2660
0.3019
0.3405
0.2904
Claude-4-somet
0.2485
0.2303
0.2539
0.2619
0.2440
0.2673
0.2844
0.2473
0.2954
Claude-4-opus
0.3044
0.2931
0.3077
0.3232
0,3103
0.3271
0.3378
0.3146
0.3447
GPTT-4o-mini
0.1306
0.1881
0.1134
0.1450
0.1976
0.1292
0.1455
0.2007
0.1290
GPT--40
0.2832
0,3334
0,2590
0.3083
0,3490
0.2861
0.3162
0.3425
0.2981
GPT-o4-mini
0.3631
0,4111
0,3463
0.3943
0.4236
0.3830
0.3791
0,4360
0.3596
GPT-o3
0.4676
0.5152
0.4533
0.4657
0.5287
0.4469
0.5328
0.5607
0.5244
Gemini-2.0-faast
0.2708
0,3020
0.2615
0.3026
0,3126
0.2997
0.2866
0,3200
0.2767
Gemini-1.5 1.S-po
0.2933
0,3854
0.2595
0.3291
0,3853
0.3053
0,3084
0.4060
0.2726
2.5-ni-2
0.4224
0.4697
0,4082
0,4548
0,4815
0.4469
0,4475
0.4963
0.4330
Domini-2.55ppro
0.4623
0.4977
0.4517
0.5044
0.5377
0.4944
0.4797
0.5150
0,4692
CMMP (0.48B)
0.4476
0.4648
0.4425
0.4077
0.4515
0.3947
0.6032
0.5742
0.6119
ADA-CM (0.44B)
0.4581
0.4257
0.4678
0.4197
0.4017
0.4251
0.6139
0.5303
0.6389
Ours (0.52B)
0.5009
0.5342
0.4910
0.4667
0,5185
0.4512
0.6443
0.6243
0.6502
Table 10: Comparison with MLLMs. Higher is better. Top-3 values in each column are color-coded
fair comparison between MLLM-based models and traditional two-stage HOI detectors, by aligning their input structure and evaluation procedure.
By clearly defining the task, unifying the input format, and standardizing the output representation, make it possible to evaluate MLLMs HOI detection in structured and consistent manner. This also allows structured predictions to be extracted from free-form text generated by different MLLMs, making quantitative comparisons across models both feasible and fair.
Evaluation Metries Since MLLMS generate textual outputs rather than full confidence scores over all possible categories, traditional metrics such mean Average Precision (mAP) are not well- suited for evaluation in this setting. Therefore, we adopt three alternative metrics: mean F1 score (mF1), mean precision, and mean recall to better reflect the accuracy of MLLM-based HOI pre- dictions. We follow the standard matching logic used in prior HOI works: predicted human-object interaction is considered true positive if and only if:
Both the predicted human and object bounding boxes have an Intersection-over Union (IoU) greater than 0.5 with the corresponding ground truth boxes;
The predicted object category and interaction (verb) label match the ground truth.
This ensures a fair and consistent evaluation for both MLLM-based and conventional HOI detection methods.
Results. We evaluate series of models under two settings: (1) R50-DETR Object Detection Prompting, where detected boxes and class labels from R50-DETR are provided as prompts; and (2) Ground Truth (GT) Object Detection Prompting, where oracle boxes and class labels are used. The compared models include MLLMS (e.g., Qwen, GPT, Gemini, Claude series) and HOI-specific methods (e.g., CMMP, ADA-CM, Ours). In the R50-DETR prompting setting, ours achieves the highest mF1 score (36.71%), outperforming both the sota HOI method baseline (CMMP, 33.24%) and all MLLMs (best being Gemini-2.5-pp 31.61%) Similar trends are observed for mPrecision and mecalll where our method achieves 33.51% and 49.82%. respectively. This demonstrates that our method is more robust under detection noise and better captures interaction semantics even
18

---

Published as a conference paper at ICLR 2026
easy hand spotial
semmatt buth
Full
Spat.
Sema.
Both
Hard
DINOv2-large Baseline v.s +LINK
37.39
65.7
24.4
48.8
16.4
DINDW2.40
41.40
58.1 4
46.51 f
51.7 f
19.4 f
CLIP-large Baseline v.s +LINK
41.2
31.4
63.0
49.3
17.0
43.3 f
42.91 f
61.4
51.7 1
19.1
Florence2 Morence2-Base-e Baseline
+IJNK
38.9
50.9
48.1
51.8
19.9
41.81
51.1 f 51.31
52.5
20.31
CLIPAF
Figure 6: Spatial vs. Semantic emphasis foundation models for HOI detection and the impact of our method.
when the input object detections are imperfect. In the GT prompting setting, which isolates the HOI prediction capability by removing detection errors, our method again achieves the highest per- formance: mF1 of 50.09%, mPrecision 46.67%, and mRecall of 64.43%. These results exceed not only MILMs (e.g., GPT-4o-mini at 36.31 %, Gemini-2.5-pra at 46.23%) but also prior HOI methods like CMMP (44.66) and ADA-CM (4)..
In particular, our method shows superior performance on Rare interactions (mFl-Rare: 53.42%) and Non-Rare interactions ImF1-N.Rarm 49.10% 49.10%) indicating both generalization ability and capacity for modeling long-tail tail HOI categories. Compared to other HOI methods, approach yields sistent improvements across all splits. Overall, these results validate the effectiveness of our unified HOI detection framework, outperforming both MILMs and previous dedicated HOI approaches under both detection-promptte and oracle-prompted settings.
A.6 SPATIAL AND SEMANTIC EMPIASES
Spatial Semantic Emphasis. To further assess the impact of proposed LINK module, investigate how foundation models vary in their spatial and semantic emphasis within unified HOI detection architecture. Specifically, compare representative baselines: CLIP-large, which is semantic-oriented due to its image-text contrastive pretraining, and DINOv2-large, which is spatial- oriented via visual-only self-supervised learning.
On the 9.6K test images (covering 33K HOI instances), we categorize samples into four subsets based on the normalized performance difference between CLIP and DINOv2: (1) Semantic- -erietic-Oriented Oriented (CLIP- - DINOv2 > 0.25), (2) Spatial-Orient (DINOv2 - CLIP > 0.25), (3) Both (CLLIP DINOv2 S 0.25), and (4) Hard cases (CLIP 0.4 and DINOv2 < 0.4).
As shown in Figure 6, DINOv2 achieves strong performance spatial dominated cases (Spat.: 65.7) but performs poorly on semantic-heavy samples (Sema.: 24.4), revealing strong spatial bias and limited capacity for high-level semantic reasoning. After integrating our LINK module, this disparity is greatly mitigated (Spat.: 58.1 vs. Sema.: 46.5), resulting in enhanced generalization across categories and clear improvement in overall mAP (from 37.4 to 41.4).
In contrast, CLIP initially demonstrates semantic preference (Sema.: 63.0 vs. Spat.: 31.4). With the addition of LINK, spatial sensitivity is substantially improved (Spat.: 42.9), while semantic strength remains largely preserved (Sema.: 61.4), leading to balanced capability and improved per- formance on hard samples (17.0 19.1).
Even for Florence2-Baan which is pretrained using multimodal multitask learning and exhibits more balanced behavior, LINK continues to deliver consistent improvements across all subsetsspatial, semantic, both, and harddemonstrating its universal adaptability and effectiveness in addressing spatial-semantic imbalance in diverse foundation models.
A.7 VISUALIZATION RESULTS
To provide a more intuitive understanding of our models behavior, we present qualitative results of HOI detection on both real and synthetic images in Figure 7.
19

---

Published as a conference paper at ICLR 2026
wid Medk 0.
(a) IIOI Predictions on Real Images (IIICO-DET Test Sec)
(b) HOI Predictions Synthetic Images (Cross-domain Zero-shot Setting)
Visualization HOI predictions on both real and synthetic images. (a) HOI detection results on real images from the HICO-DET test set demonstrate strong instance awareness and pre- cise interaction reasoning.
Figure 7:
(b) Cross-domain zero-shot predictions on synthetic images highlight the models generalization ability and robustness to visual domain shifts.
As shown in Figure 7(a), on real-world test images from HICO-DET, our method demonstrates strong instance awareness and precise HOI reasoning. For example, in the second image of the first row, our model accurately identifies the action feed for all relevant bird instances (birdObird4), while also distinguishing the hold action exclusively for birdO. Notably, it correctly predicts no interact i on for the background birds, showcasing its ability to selectively recognize meaning- ful human-object pairs amidst complex scenes. In the fourth image of the same row, our model successfully disambiguates multiple overlapping interactions involving the same object. It predicts that both the man and the woman are sit on the bench, while additionally identifying the wom- distinct HOI triplet. This demonstrates the
ans action of hold ld and talk models capacity to capture layered and concurrent human-object relations in everyday scenes.
cellphone
In Figure 7(b), we further demonstrate the models zero-shot generalization synthetic and out- of-domain images. Despite visual domain shifts, our model continues to detect plausible HOI triplets (e.g., ride horse, brush with toothbrush, use cell phone) with high fi- delity, highlighting its cross-domain robustness and ability to generalize to unseen scenarios.
A.8 PROSS-DOMAL EVALUATION
In this section, evaluate the cross-domain generalization ability of our method and compare it against previous methods. As illustrated in Table 11, we consider multiple domain shift settings, where the domain the left side of the arrow indicates the training domain, and the right side denotes the testing domain. Specifically, SWiG, H. and V refer to the SWIG-HOI, HICO-DETT and V-COCO datasets, respectively. For example, SWIG
HICO indicates that the model is trained
20

---

Published as a conference paper at ICLR 2026
Method
|SWIG- SWIG-HICO-DETT SWIG-V-COCO HHCO--EET--CCCOCCCCCCCCC Real Syythetiii
PVCC
9.73/ 13.66
38.2
45.1
/1.32
MP HO-
44.2
ADA-CM
46.6
29.87/ 28.17
CMMP
29.43 28.52
Ours
26.64/ / 30,95
51.2
$2.8
35.91 34,91
Table 11: Cross-domain HOI detection results. Each entry reports mAP scores (Full / Rare when available) on the target domain without fine-tuning. Our method consistently outperforms prior works across all cross-domain settings.
on SWiG-HOI and directly evaluated on HICO-DET without any fine-tuning. The setting Real -> Synthetic evaluates the model trained on real-world images and tested on synthetic/generated ones, with partial visualization results presented in the previous section (Figure 7).
These results highlight the superior cross-domain These results underscore the strong cross-domain generalization capability of our approach. Across all evaluated settings, our method achieves sub- stantial improvements over prior works. Notably, in the SWiG-HICO setting, our model signifi- cantly outperforms PVIC (26.64 vs. 9.73 on Full, and 30.95 vs. 13.66 on Rarc), demonstrating the high transferability and generality of the learned representations.
In the HICO-vV setting, despite both datasets containing real-world images with differing verb model achieves 52.8 mAP, surpassing the previous best of 47.2 by CMMP. This highlights the effectiveness of our HO Geometrical encoder and VLM Linking de- coder in capturing robust, transferable features across varying interaction patterns. Furthermore, in the Real--Syaliieeeeee SyytheSynthetic scenariowhere substantial shifts in visual appearance, object style, and contex- tual cues existour model maintains strong performance (35.91 34.91), outperforming all baselines. These results collectively demonstrate the robustness and adaptability of our framework under di- verse and challenging domain shifts.
and object distributions,
How to Generate Synthetic Images with Annotations. In the evaluations reported in Table 11, we use the original test sets SWiG-HOI, HICO-DET, and V-COCO for the respective settings. However, for the Real Synthetic evaluation, we need to generate synthetic images with corre- sponding HOI annotations to enable quantitative assessment.
To this end, adopt InteractDiffusion Hoe et al. (2024), condition-controlled image generation framework based on Stable Diffusion that extends existing pre-trained text-to-image (T2I) diffusion models to better incorporate human-object interaction conditioning. Specifically, use the Inter- actDiffusion model built upon the Stable Diffusion XL version. We take the bounding boxes from the HICO-DET test set layout guidance and use their corresponding ground-truth HOI triplets as text prompts. These two signals, layout and text, are jointly fed into model to generate realistic images that reflect the specified interactions. This approach allows us synthesize new test set of images with known annotations, enabling systematic evaluation of HOI models under domain shifts from real synthetic data.
A.9 COMPUTATIONAL COMPLEXITY ANALYSIS
Our training pipeline involves an additional pre-training stage for a teacher model, followed by stu- dent training supervised by both ground-truit annotations and teacher guidance. While this design raises potential concerns regarding computational resources and training time, we empirically show that these costs remain within reasonable range.
As illustrated in Figure 8, our method achieves competitive performance within just one epoch and reaches full convergence in 15 epochs, significantly faster than prior methods in terms of conver- gence speed. Thanks to this rapid convergence, our method requires fewer total epochs. Although the overall training time is slightly longer due to the inclusion of teacher supervision, the increase is acceptable given the performance gains. Moreover, our method maintains computational effi- ciency during inference. For instance, LINK-B achieves 37.43 mAP with only 80.1 GFLOPs. Since the teacher-studen paradigm is used solely during training, does not incur any additional cost at inference time, ensuring that the model remains efficient.
21

---

Published as a conference paper at ICLR 2026
Method
Perf
GFLOPs
Times
42-
43.36
UPT
31,66
60.9
3h47m
40
39,82
PVIC
34.69
62.1
4h02m
CMMP
33.24
114.0
3h24m
16:10
36.35
CMMP
38.14
168.0
6h18m
36-
54,
LINK B
37.43
80.1
4h16m
34
LINK-B -
39.43
79.9
4h56m
LINK-L 43.25
252.5
6h48m
30
11:27
31.39
LINK-L* 43.82
251.9
6h04m
CHNP
Teacher model require an additional
28
CHPP
ADA-UII
one-time cost of 4h12m.
10
15
20
(a) mAP V.S. Epuchs.
(b) Comparison with SOTAs.
Figure 8: This illustrates the performance progression over epochs and the total time required for training. All experiments are conducted on four NVIDIA-RTX-4090 GPUs.
In summary, our method converges faster and delivers a favorable balance between training cost and final model efficiency.
A.10 SCALABILITY
We visualize the performance of various methods across different computational scales in Figure 9, plotting mAP against GFLOPs on logarithmic axis.
Color
Ours
SwinL-ViT-G
Ours
CMMP
Ours
50
SWiTL
PVIC
CLIP4HOI
Near- -inearr Trend
UPT
FZ-HOI
EZ-HOI
Ours R50 VIT-L
PVIC-Swin
SwIaL-VIT-L
CMMP
(%) dvW
SwiuL-ViT-L
Marginal effect
40
R5T-VIT-R
Ours
R50-VTT-L
CMMP
E7-HOI
PVIC-R101
PVIC-R50
35
CMMP RSO-V
R50:VIT-L
EZ-HOI
R50-VIT-B
UPT-R50
UPT-RI01
GFLOPs (log scale)
50
100
500
1000
2000
Figure 9: Our method exhibits nearly linear performance growth with respect to GFLOPs under the logarithmic scale, demonstrating excellent scalability. Moreover, across all model sizes, our method consistently achieves state-of-the-arr performance compared with existing methods.
Our method demonstrates exceptional scalability, exhibiting an almost linear performance trend (yellow dashed arrow) computational capacity increases under log-scale GFLOPs. When equipped with a Swin-Large backbone and further scaled via a ViT-Giant VLM, our LINK model reaches 50.5 mAP HICO-DET dataset. markedly outperforming all existing methods.
In contrast, VLM-basee methods such as CMMP and EZ-HOI display progressively diminishing performance gains model size increases (blue/green dashed arrow), deviating substantially from
22

---

Published as a conference paper at ICLR 2026
linear scaling and revealing clear marginal returns. Moreover, MonVLM-basee pipelines such as UPT and PVIC are unable to leverage large-scale pretrained models to effectively scale up.
These results collectively underscore the superior scaling characteristics of our method, enabling consistent and efficient utilization of expanding computational and representational capacity.
A.11 PLUG-AND- WITH OPEN- VOCABULARY DETECTORS
HICO-DET (Default)
HICO-DET (Known-Object) HICO -> V-COCO
Open-vocabulary Detectors
Full
Rare
N-rare
Full
Rare
N-rare
AP
YOLO-world
27.07
29.83
26.24
30.06
32.37
29.37
33.61
YOL.-world-m
30.35
33.40
29.45
33.39
36.00
32.62
37.31
YOLO-world-I
32.64
36.51
31.49
35.98
40.06
34.76
39.28
YOL.O-worldx
33.73
37.30
32.66
36.98
40.65
35.88
40.35
Grounding-DIne DINO swin-tiny
32.97
38.21
31.40
35.96
40.76
34.52
36.56
Grounding-DINe swin-base
39.69
45.99
37.81
42.42
47.94
40.77
46.55
Qwen3-VL-8B
39.61
45.05
37.99
42,02
47.37
40.42
Table 12: Comparison of various OVD models on HICO-DET (Default and Known-Object settings) and cross-dataset transfer from HICO-DET to V-COCO.
We conduct extensive evaluations across several state-of-the-art open-vocabulary detectors (OVDs), including YOLO-Worke (s/m/l/x), Grounding- DINO (Swin- (win-Tiny/Base),, and Swen3-VL--BP None of the detectors are fine-tuned on the target dataset, and our LINK model is also not adapted to any detector, making the entire pipeline fully training-free and plug-and-play.
-rOOOding-DINO
For fair comparison, we adopt the following configurations:
Yorl-World: input resolution 640 and score threshold 0.25.
Grounding-DING box threshold of 0.35 and text threshold of 0.25.
QW. top_p=0.8, k=2, temperature=0.2, max tokens=2048; outputs are nor- malized to j0, 1000].
Despite being entirely training-free, our method achieves strong and consistent performance across all evaluated OVDs. Notably, combining LINK with Mrounding-DINO (Swin-Bass) even sur- passes several approaches that rely on fine-tuned detectors.
We further evaluate the HICO + V-COCO transfer setting, where LINK model trained on HICO- DET directly paired with each OVD. Our method remains robust under this cross-dataset setting, demonstrating the scalability and strong generalization ability of our design.
We also present qualitative comparisons illustrating how different open-vocabulary detectors behave in real scenes and how LINK leverages their outputs for HOI reasoning. The first three columns vi- sualize the raw detection results from Wwen3-VL-BB. YOLO- World-X, and GroundingDINO Swin- Base, respectively. These detectors often generate large number of candidate boxes with hetero- geneous confidence distributionsan inherent challenge when performing HOI reasoning in fully training-free setting.
Despite these inconsistencies, the last column shows that LINK produces accurate and semantically coherent HOI predictions when directly paired with GroundingDINO Swin-Base, without any fine- tuning on either detector or HOI model. Notably, in the third row, the scene contains many irrelevant object proposals (e.g., multiple cars in the background and a baseball glove near the player), yet LINK successfully filters out distractors and identifies the correct human-object interaction. It out- puts meaningful interaction labels such as hold, wield, and swing for the baseball bat, demonstrating the robustness and strong generalization capability of our method under noisy OVD detections.
A.12 ROBUSTNESS UNDER INACCURATE DETECTED BOXES.
To evaluate architectural robustness, we introduce two forms of spatial perturbation: box-shift and box-scale. For box-shift, bounding box with width w, height h, and center (Cz, Cy) is perturbed by uniformly sampling a new center within [C, + dw] and [C,, + 5h.. For box-scale, the width and height are randomly rescaled within the range [(1 8), (1 5)). We evaluate 8 E [0, 1). and
23

---

Published as a conference paper at ICLR 2026
Owen3-VL--8
YOLO-Wardd-X
GroundingDINO Swin-Base
LINK Prediction (itaed en CimendingOONN)
Figure 10: Qualitative results comparing OVD detections (first three columns) with HOI predic- tions produced by LINK using GroundingDINO detections (last column). Although OVDs output numerous proposals, LINK reliably identifies the correct human-object pairs and interactions.
Perturbation Strength
Perturbation Type
0.0
0.2
0..4
0.6
0.8
1.0
Full
42.93
42.69
42.25
41.03
39.41
38.05
Box-Shift
Rare
45.03
44.60
44.08
42.95
41.33
40.30
Non-Rare
42.20
42.02
41.61
40.35
38.73
37.27
Full
42.93
42.81
42.33
41.49
40.03
38.30
Box-Scale
Rare
45.03
44.89
43.92
42.92
41.03
39.10
Non-Rare
42.20
42.09
41.78
40.96
39.63
37.97
Table Robustness analysis under spatial perturbations. report mAP for Full, Rare, and Non- Rare subsets across different perturbation strengths for both box shifting and box scaling.
the corresponding results are summarized in Table 13. Even under severe perturbation (e.g., 8 1.0), our method maintains over 38.0 mAP, which is comparable to the undisturbed performance of several state-of-the-a approaches (e.g.. CMMP, ADA-CM. EZ-HOI). Under mild perturbation (c.g.. 0.2), the performance degradation negligible (less than 0.3 mAP). Redundant detections not concern in our setting, they can be reliably handled using standard NMS suppression.
This robustness stems from our VLM Linking Decoder, which does not rely solely on geometric cues ROI features. Each decoder layer attends to the global VLM representation, which remains stable even when box coordinates are perturbed, enabling reliable HOI reasoning. As illustrated in Fig. 1, small boxes perturbations do not affect the predictions of our LINK model.
24

---

Published as a conference paper at ICLR 2026
Original
Shifted
Scaled-Dowa
Scaled-Up
Figure 11: Visualization of bounding-bar boxx perturbations: Shifted, Scaled-Down, and Scaled-Up.
Figure 12: Robust HOI prediction under redundant detection boxes.
Moreover, the plug-and-play experiments with diverse open-vocabulary detectors (Table 12) fur- ther support this conclusion from complementary perspective: although different OVDs pro- duce bounding boxes that vary greatly in confidence distribution, density, and localization bias, our method remains fully training-free, plug-and-play, and consistently strong across all detectors.
25

---

Published as a conference paper at ICLR 2026
A.13 INSUFFICIENT SPATIAL UNDERSTANDING CLIP
(a) Layer-wise PCA Visualization of CLIP-ViT-L/1403366
Lapar
Lager 5
34
(b) Layer-wise PCA Visualization of DINOv2-base
Laye 8
Lape
Leow
Lawer
Layer 32
Lyyer 11
Figure 13: Layer-wise PCA visualization of patch embeddings. CLIP V. DINOv2
Similelly
Feature2
3.890
0.122
2000
5.0033
0,0057
05990
05825
05990
04835
6.379
8.000
LOKE
DIROS
2.003
0,000
05890
Dialance
0,25
0.44
80000
0982
5.2200
0,000
6.0533
0,20
ROI Featurel ROI Featurch ROI Featurcs
KGD
8:02
-0.00
Figure 14: CLIPs feature-map tokens exhibit insufficient spatial variability.
26


Reviews:
[{"review_id": "F56MRWBrwE", "reviewer": "Reviewer_NNji", "summary": "This paper proposes a unified framework for HOI detection. It introduces two key components, a Geometrical Encoder, which models spatial relationships between humans and objects, and a VLM Linking Decoder, which bridges global vision-language features with instance-level HOI reasoning. In addition, the paper presents a Progressive Learning Strategy based on a teacher-student paradigm, delivering dense supervision to all human-object pairs to alleviate sparse annotations. The experimental study is comprehensive, demonstrating consistent improvements across fully supervised, zero-shot, few-shot, and open-vocabulary settings on multiple benchmarks.", "questions": "Please refer to the weaknesses. No other questions.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. A well-motivated design balancing generalization to unseen HOIs and HOI-specific reasoning. The proposed Geometrical Encoder and Linking Decoder jointly enhance HOI reasoning while preserving model generality across diverse foundation models.\n2. The Progressive Learning strategy provides dense supervision via teacher-student distillation, enabling the model to learn from all human-object pairs and effectively alleviate sparse-annotation issues.\n3. The experiments are comprehensive and achieve SOTA performance consistently.", "weaknesses": "1. The motivation for introducing the Geometrical Encoder is not clearly justified. The paper claims it enhances spatial awareness, yet it lacks quantitative or cited evidence showing insufficient spatial understanding in existing VLMs like CLIP. \n2. In Table 6, the baseline (A1) used for ablation is under-specified. It is unclear what feature representation or decoder architecture it employs. When evaluating the proposed Geometrical Encoder and VLM Linking Decoder, the paper does not clarify whether the baseline uses the original spatial encoding or a standard decoder. This ambiguity makes it difficult to assess the exact improvement source.\n3. The teacher model in the progressive learning strategy uses a stronger ViT-L backbone to supervise the ViT-B student, introducing extra knowledge capacity beyond dense supervision. A comparison using a same-scale teacher (e.g., ViT-B to ViT-B) is necessary to ensure the fair comparison.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "sB2WxdDlR1", "reviewer": "Reviewer_duFB", "summary": "To address the sparsity of supervision and the absence of a generalizable architecture, this paper propose a HOI detection framework equipped with a Human-Object Geometrical Encoder and a VLM Linking Decoder, and develop a Progressive Learning Strategy under a teacher-student paradigm to deliver dense supervision. Extensive experiments demonstrate state-of-the-art results in both zero-shot and fully supervised settings.", "questions": "See the limitations and cons above.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. The paper provides a well-articulated motivation, perceptively identifying two main challenges in HOI: (1) the sparsity of supervision, and (2) the absence of a generalizable architecture capable of achieving strong performance in both fully supervised and zero-shot scenarios. In response, the paper propose a novel learning strategy and a method to address the limitations.\n\n2. The proposed Human-Object Geometrical Encoder and VLM Linking Decoder comprehensively account for both spatial and semantic associations between HO pairs, maintaining consistent performance across diverse settings. The learning learning strategy, where the student model is jointly supervised by explicit ground-truth annotations and guidance from a pre-trained teacher, effectively enhances its generalization capability.\n\n3. Comprehensive experiments compare the proposed model with baselines across zero-shot, few-shot, fully supervised, and open-vocabulary scenarios. The results demonstrate that LINK excels in both specialization and generalization.", "weaknesses": "1. Risk of Error Accumulation in Model Architecture: inaccurate detector outputs (e.g., incomplete bounding boxes, redundant detections) can lead to erroneous geometric relationship, erroneous position encoding, and imprecise ROI feature extraction, thereby adversely impacting subsequent inference steps. The paper lacks experiments or visualizations to substantiate the robustness of the proposed model under such conditions.\n\n2. In the application, detectors are fine-tuned on target dataset rather than functioning as a training-free plug-and-play module. It is recommended to explore stronger open-vocabulary detectors, such as Yolo-World, Grounding-DINO.\n\n3. While knowledge distillation can enhance generalization ability, the teacher and student models in this work share identical architectures and capacities. Such a distillation setup may not yield substantial improvements and could potentially degenerate into a simple continued training process.\n\n4. Insufficient Baseline Comparison. Sseveral state-of-the-art baselines (e.g., BC-HOI, UniHOI, LAIN) are not included for evaluation.\n\n5. The corner-mark annotations in Tab.1 left unexplained.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "o8r6W7gfGv", "reviewer": "Reviewer_EQVH", "summary": "This paper proposes LINK, a method for Human-Object Interaction (HOI) detection that addresses the specialization-generalization trade-off. Its core is a decoupled architecture with a geometrical encoder and a vision-language model (VLM) linking the decoder, enabling plug-and-play use with any object detector. To overcome sparse supervision, it introduces a progressive learning strategy using self-distillation, which provides dense, instance-level guidance across all human-object pairs. The method achieves new state-of-the-art results across fully-supervised, zero-shot, and few-shot benchmarks, demonstrating strong generalization and open-vocabulary capability.", "questions": "Please kindly refer to the weakness section.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- This paper investigates the critical challenges in HOI detection: the sparsity of supervision.\n- The experimental results demonstrate the effectiveness of the proposed methods.\n- The evaluation covers various settings, including zero-shot, few-shot, and fully-supervised HOI detection.", "weaknesses": "- The proposed teacher-student paradigm has an inherent circular dependency. The teacher model, which is intended to mitigate annotation sparsity by providing dense supervision, is itself trained solely on the original sparse annotations. This bootstrapping approach may limit the upper bound of knowledge that can be distilled to the student, as the teacher's guidance is fundamentally constrained and potentially biased by the initial sparse data. \n- The teacher-student paradigm lacks significant novelty. Knowledge distillation is a well-established and generic technique in deep learning. While its application to the HOI domain is sensible and effective, the paper does not specifically optimize the distillation for HOI problem, but rather adapts an existing general-purpose framework (using logit, feature, and query-level losses) to a new problem.\n- The architectural generality is not a distinct advantage over existing paradigms. The paper positions the lack of a generalizable architecture as a key challenge. However, numerous prior VLM-based methods (e.g., those relying on CLIP embedding matching for relation prediction) are inherently designed to operate in both fully-supervised and zero-shot settings by leveraging the same pre-trained semantic space. Therefore, the goal of creating a generalizable architecture is not novel, and the paper's claim rests primarily on its specific implementation (the LINK framework) achieving superior performance, rather than on identifying a new problem.\n- Missing explanation of annotation in the Table. In Table 1, the subscript on the LINK is not explained, which causes confusion.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)