Zero-shot HOI Detection with MLLM-based Detector-agnostic Interaction Recognition
Authors: Shiyu Xuan
Dongkai Wang
Zechao Li
Jinhui Tang
Venue: ICLR 2026 Poster (2026)
Keywords: Human-Object Interaction; Large Multi-modal Language Models;
Primary area: applications to computer vision, audio, language, and other modalities

Abstract: Zero-shot Human-object interaction (HOI) detection aims to locate humans and objects in images and recognize their interactions. While advances in open-vocabulary object detection provide promising solutions for object localization, interaction recognition (IR) remains challenging due to the combinatorial diversity of interactions. Existing methods, including two-stage methods, tightly couple IR with a specific detector and rely on coarse-grained vision-language model (VLM) features, which limit generalization to unseen interactions. In this work, we propose a decoupled framework that separates object detection from IR and leverages multi-modal large language models (MLLMs) for zero-shot IR. We introduce a deterministic generation method that formulates IR as a visual question answering task and enforces deterministic outputs, enabling training-free zero-shot IR. To further enhance performance and efficiency by fine-tuning the model, we design a spatial-aware pooling module that integrates appearance and pairwise spatial cues, and a one-pass deterministic matching method that predicts all candidate interactions in a single forward pass. Extensive experiments on HICO-DET and V-COCO demonstrate that our method achieves superior zero-shot performance, strong cross-dataset generalization, and the flexibility to integrate with any object detectors without retraining. Code will be released.

Published as a conference paper at ICLR 2026
ZERO-SHOT HOI DETECTION WITH MLLM-BASED DETECTOR-AGNOSTIC INTERACTION RECOGNITION
Shiyu Xuan', Dongkai Wang?, Zechao : Tang*
School of Computer Science and Engineering, Nanjing University of Science and Technology
2 School of Computing and Artificial Intelligence, Southwestern University of Finance and Economics
a Nanjing Forestry University
shiyu_xuan @ njust.edu.cn, wdk @ www.fe.dd..cn zechao. cijtao.li@nuus.cddu... tangjh @ njfu.edu.cn
ABSTRACT
Zero-shot Human-object interaction (HOI) detection aims to locate humans and objects in images and recognize their interactions. While advances in open- vocabulary object detection provide promising solutions for object localization, interaction recognition (IR) remains challenging due to the combinatorial diver- sity of interactions. Existing methods, including two-stage methods, tightly cou- ple with a specific detector and rely on coarse-grained vision-language model (VLM) features, which limit generalization to unseen interactions. In this work, we propose a decoupled framework that separates object detection from and leverages multi-modal large language models (MLLMs) for zero-shot IR. We in- troduce a deterministic generation method that formulates IR as a visual question answering task and enforces deterministic outputs, enabling training-fres zero- shot IR. To further enhance performance and efficiency by fine-tuning the model, we design spatial-aware pooling module that integrates appearance and pair- wise spatial cues, and a one-pass deterministic matching method that predicts all candidate interactions in a single forward pass. Extensive experiments on HICO-DET and V-COCO demonstrate that our method achieves superior zero- shot performance, strong cross-dataset generalization, and the flexibility inte- grate with any object detectors without retraining. The codes are publicly available at https://gihhub..ooSS..... Suamm//SYXun//AHHHHA
INTRODUCTION
Human-object interaction (HOI) detection aims to localize humans and objects in an image and rec- ognize the interactions between them. provides a fine-grained understanding of human activities, which is crucial for downstream applications such as robotic manipulation (Shridhar et al.) [022]] Jin et al. 2024), image captioning (Wu al. 2022 Yao et al. 20188, and autonomous driving (Liao) et al. 2025 Wang et al. 2023). Despite recent progress, HOI detection remains challenging due to the large combinatorial space of human-object pairs and the need for fine-grained visual understand- ing. Moreover, real-world applications often require recognizing unseen human-object interactions that do not appear in the training set, leading to the task of zero-shot HOI detection.
To address zero-shot HOI detection, recent works (Ning et al. 2023 Lei et al. 2025 2024b 2023)
exploit languael models such as CLIP (Radford et al. 2021) to transfer semantic knowledge from language HOI detection. Typically, they use embeddings of interaction descriptions, e.g. "A person is holding a cup". " to construct classifiers for unseen interactions. While effective to some extent, these methods face limitations. First, CLIP features lack the fine-grained represen- tational capacity required to distinguish visually similar interactions. Methods have to incorporate detector features to obtain fine-grained representation of instances. Second, their core mechanism, aligning visual and text features, operates only on categories observed during training, which re- stricts generalization to unseen interactions. Progress in open- vocabulary object detection (Liu et al.) 2024 Cheng et al., 2024) provides sufficient solution for localizing unseen objects, while interac- tion recognition (IR) remains the main bottleneck. As shown in Fig. (a), most existing methods,
"Corresponding Author

---

Published as a conference paper at ICLR 2026
Image
Intoraction Descriptions
VLM Embeddings
Image
Quostion Prompt
Parson nding bicycle
< fing > The festures of Intoraation we </itee Salert the cormet intrrattion riding Noydeesooips holding a bicycledao... siting on a licydespull.
Person bolding bicycle
VLM
Object
Interaction Encoder
Object Detector
MLLM
D Detector
Human- Object Pair Detector Features
Interaction Embeddingsi
riding boydde Interaction Recognition
D Detection Results
Coupled with specific detector
Decouple object detection and interaction recogmition
Pour general onation detector footures
Plug and play integration with any detector Surong cruss modal generalication of MLLMs
Coasse graned VLM Jestures
(a) Existing HOI Detection Methods
(b) Our Method
Figure 1: (a) Existing methods, including two-stage methods, couple object detection and inter- action recognition together. Their performance are constrained by the limited generalization of detector features and coarse-grained VLM features. (b) Our method fully decouples these two pro- cesses and harnesses the powerful MILMS for interaction recognition. This design benefits from the generalization of MLLMs and advanced detectors.
including two-stage methods (Mao et al. 2024 Lei al. 2025 Kim et al. 2025 couple IR with a specific object detector (Carion et al.] 2020). These methods rely on detector features or inter- object relations to enhance VLM features with fine mne-grained representational capacity, but struggle to balance fine-grained detector features with generalized VLM features. As a result, they limit inde- pendent improvements to IR and detection, and the detector cannot be changed without retraining.
To overcome these limitations, we propose a decoupled framework that separates object detection from IR. Effective IR requires representations that are both fine-grained and broadly generalizable. Unlike CLIP-based methods that rely on static embeddings, multi-modal large language models (MLLMs) are trained on large-scale image-text pairs and instruction-follot following tasks, equipping them with strong cross-modal generalization. This motivates us to exploit MLLMS for IR in zero- shot HOI detection. By decoupling object detection and IR, our framework enables plug-and and-play integration with any object detector, allowing to focus on advancing IR without being constrained by the training, architecture, and limited generalization capacity of the detector.
Building on this foundation, for each human-objen pair, we formulate IR as visual question answering (VQA) task, where both the pair information and candidate interactions are encoded into prompts for the MLLM. To adapt the open-ended text generation of MLLMs to the multi-label classification setting of IR, we introduce a deterministic generation method that restricts the answer within the candidate interaction list, establishing strong training-free IR method.
Despite these improvements, two challenges remain. First, the features extracted from human and object bounding boxes are sensitive to imperfect detections and fail capture the pairwise spatial information crucial for IR. Second, deterninistic generation requires multiple forward passes for IR, resulting significant computational overhead when the candidate list is large. To address these issues, we propose a spatial-aware pooling module that incorporates both appearance and pairwise spatial cues, and one-pass deterministic matching strategy that reformulates generation as feature matching, allowing all candidate interactions to be predicted within single forward pass. Fine- tuned on a training set, these components further enhance both performance and efficiency.
We evaluate our method on HICO-DET and V-COCO, demonstrating superior performance com- pared to existing zero-shot methods, while also validating strong cross-dataset generalization, egg., outperforming CMMP by 12.26%. This is early work to fully decouple the object detection and IR in HOI detection. By integrating the MLLM for IR, our deterministic generation method enables triining-free IR, while the proposed spatial-aware pooling and one-pass deterministic matching fur- ther improve fine-tuning performance and inference efficiency. Our framework achieves promising results in both zero-shot and cross-dataset settings. Once trained, method has the flexibility to integrate with any object detector without retraining, establishing a new paradigm for HOI detection with M...Ms.
2

---

Published as a conference paper at ICLR 2026
2 RELATED WORKS
Human-Object Interaction Detection. HOI detection methods are generally divided into two cat- egories: two-stage and one-stage. Two-stage methods follow
werectioontten-rroggninn pipeline, where humans and objects are first localized and then paired for interaction recognition. Early works such as InteractNet (Gkioxari et al. 2018) and iCAN (Gao et al., 2018) enhance appearance mod- eling through human-centric or attention-based features. Beyond appearance cues, some methods integrate spatial cues and structural modeling via GNNs or attention mechanisms (Gao et al. 2020
Qi et al.] 20188] Zhang et al. 2022a; Ulutan et al., 2020 Zhang et al., 1021 2023), while others
exploit human pose or part-level features (Moon et |021 Wan et al. 2019 Zhou & Chi 2019).
One-stage methods directly predict numan-object--neeee
triplets. Point-based methods (Liao
et al.
2020:
Wang et al.] (2020) reformulate pairs as keypoints, while query-based methods (Kim
et al.
2021 Zhou al. 2022 Zou al., 2222 Chen et al.] 2021] Tamura et al., [021] leverage
query-based decoding to predict triplets end-to-end. Follow-up works further improve performance
through query optimization (Dong al.
2021 2022), large-scale pre-training (Li al. 2024b). and
structural enhancements (Zhang et al. 20226 Kim et al. 2022; Tu et al. 2022
Zero-shot HOI Detection. Conventional methods fail to generalize to unseen interactions. Early attempts (Hou et al., 2020; (021) disentangle object and verb features for compositional general- ization, but cannot handle unseen objects or verbs. With the emergence of languae models (VLMs) such as CLIP (Radford et al.] [021], recent methods exploit text embeddings of interaction descriptions for semantic transfer. For instance, VLKT (Liao et all]2202)) aligns interaction fea- tures with CLIP embeddings, while HOICLIP (Ning et al. 2023) enhances fine-grained recognition via verb adapters. ADA-CM (Lei et al., 2023) introduces concept-guided memory for training-fres or fine-tuning scenarios. LAIN (Kim et al. 2025) improves locality and interaction awareness.
Leveraging MLLMS for HOI Detection. To overcome the limited granularity of CLIP features, several works explore LLMs and MLLMs. CMD-SE (Lei et al. (024bb and UniHOI (Cao et al. 2023) generate textual descriptions of interactions, while BC-HOI et al. 2025) employs MLLM- based captioning loss to supervise training. Other efforts, such RLIP (uaan al.) 2022 2023) and MP-HOI (Rombach et al. 2022), expand training data via relational pretraining or generative augmentation. Although these method exploit auxiliary signals from LLMs or MLLMs, they still rely heavily on CLIP features and remain entangled with specific detectors.
We depart from prior work by fully decoupling interaction recognition from object detection. In- stead of relying on CLIP embeddings, we directly leverage MLLMs for interaction recognition. By reformulating the task as VQA with proposed deterministic generation, our framework enables training- free zero-shot prediction. Furthermore, our spatial-awar aware pooling and one pass determinis- tic matching methods explicitly address robustness and efficiency issues through model fine-tuning, setting our method apart from both CLIP-based and MLLM-based prior works.
METHODOLOGY
3.1 OVERVIEW
Given an image I, HOI detection aims to locate humans and objects, and recognizes the interactions (verbs) between each human-object pair. Conceptually, the interaction can be defined as a triplet (B,, B,, (A, C.)), where B₁ and B, denote the bounding box of the human and object, respec- tively, C. E C (c1, c2...... CN.) denotes the object category, and A & {ai, a2 aN denotes the interaction category. Traditional HOI detection assumes the C, A, and their combina- tion appear in the training set. Differently, zero-shot HOI detection, some combinations object and verb (Unseen Composition, UC), some categories of objects (Unseen Object, UO), and some categories of verbs (Unseen Verb, UV) would disappear.
The progress in vocabblcyy object detection (Liu al.) 2024 Cheng al. 2024) provides promising solution for object localization in HOI detection. In contrast, interaction recognition (IR), especially for unseen interactions, remains challenging task. To alleviate the above restrictions, fully decouple the object detection and IR, and leverage the powerful Multi-modal Large Language Models (Liu et al.) 2023 Bai et al. 2025) (MLLMs) for zero-shot IR. Specifically, given detection results of any detector {C", B where C* denotes the object category, we associate every human
3

---

Published as a conference paper at ICLR 2026
Linteractivennes
Ofjertt Detectare
MLP
Spdiia Aware
Rodligg
EROW owwenfireeeeee p
Cross A thention
Spatial Embbekinng
Q
K
V
Intoraction
Feature
Juse
Phoxter
Visuel
MLP
Feature Map
<fing The interaction Teatures <finer Select the conrect interaction from the list: beartlngg carsholls, drivigg earchotjp, inspecting cardhoijp.
B
ROI
7w:00
Align
Question Prompt Input
Image Feature Map
Overall
(b) Spatial-Aware Pooling
The overall framework of our method and the spatial-aware pooling (SAP). (a) The pro- posed method decouples the object detection and interaction recogmition for HO1 detection. With the detected human-object pair, MILM is used recognize their interaction. To enhance both performance and inference efficiency, SAP integrates appearance and pairwise spatial cues, and a one-pass deterministic matching method enables the prediction of all candidate interactions in a sin- gle forward pass. (b) SAP takes the human and object features as input. The cross attention layer aggregates features beyond the area bounding box, enhancing robust to the noise in the detection results. Spatial Embedding encodes the useful pairwise information into the interaction features.
Figure 2:
instance with each object instance to formulate the human-object pairs,
(Bi, B Ct N + (B', C,, B* C* 1i + k. C human bumaa))==.NNall=-.......
(I)
where Cj the object category of the i-th human-object pair. For each human-object pair, MLLM used predict the confidence score { indicates the probability of the human-object pair having corresponding interaction, M is the number of interaction categories. In the following parts, we will first introduce the MLLM-based training- free IR, and further propose tuning method to break through the bottlenecks of MLLMs for IR. The overall framework is shown in Fig.
Name (MILM((B), B C Nim where St. E RM
3.2 MLLM-BASED TRAINING- FREE INTERACTION RECOGNITION
MLLMs are constructed with visual encoder fimg = (v) which extracts the features of a given image, and an LLM. MILMS employ visual question answering (VQA) to address various tasks: Given question prompt Q and Sing. they are formulated as the prefix input for the LLM to generate the answer. The final result can be obtained through parsing the answer. To cast IR as a VQA task, given an associated human-object pair (B, Bi C.)). we construct question prompt Q' that asks the MILMM to recognize the interaction:
Question: <img^> The interaction features are < Kiner Select the correct interaction from the list: <0(C))>
The <fin>> is replaced with fing- To guide the MLLM focus on a specific human-object pair, extract human and object features through ROIAlign (He al. 2017) and concatenate them construct interaction features:
fh. f: ROIAlign(fingg [B], Bt)),
(2)
f Concat(fi, (fi,
4

---

Published as a conference paper at ICLR 2026
To achieve open-vocabulary IR, we define the candidate interaction list ((C)) based the category of detected object, feeding a bird, chasing bird, holding a bird. This list gives a hint to prompt the MLLM to select the correct interaction, e.g., e.g.. feeding bird, holding a bird.
Deterministic Generation. Although MLLM shows strong generalization ability for various tasks, directly using the above pipeline leads to poor performance. The reasons are: a) IR is multi-label classification task, i.e., a human-object pair has multiple interactions. b) The metric mAP requires the model to give a confidence score to the prediction. The model can give answers with various format and tends to generate one interaction even after supervised fine-tuning as shown in Table 3 Moreover, the confidence score hard obtain.
To overcome these limitations, we change the open open-ended text generation of the MLLM to determin- istic generation proxy task. It leverages the discriminative capability of MLLMs by estimating the semantic similarity between the question prompt and the candidate interaction. Specifically, given the candidate interaction list of the detected object in text format: (C))) = {T1, T TM ). we measure the semantic similarity between the prompt question and the interaction by calculating the conditional likelihood for the MLLM to generate the corresponding interaction.
N
ST - P(T:4, = II p( ==1
1,2, M,
(3)
where Q' is the question prompt, t[j] is the j-th token in the answer Tu. and S([k] is the semantic similarity between Q' and the interaction Tk. which can he used the prediction confidence score. Deterministic generation prevents the MILMM from generating answers unrelated to the interaction. As shown in the experiments, this method significantly improves the performance in a training -free manner. To further improve the performance, we can perform supervised fine-tuning on the dataset. Note that deterministic generation is still essential.
3.3 SPATIAL-AWARE POOLING
The proposed training-free IR method exploits the strong generalization ability of MILMMs and achieves competitive performance. However, there are still some bottlenecks: a) The interaction features are obtained with ROIAlign based on the detected bounding box. The pooling features are restricted within the bounding box, making it sensitive to the noise in the detection results, e.g. the bounding box may cover part of the object or include some background. Moreover, it ignores the human-object pairwise spatial information that is proved to be important for IR (Zhang et al. 202T 2022a). b) The computational complexity is high. Calculating the confidence score of interaction requires a forward pass as in Eq. 3 Given an interaction list (Ti, T ,2. with M candidate interactions, M forward passes needed.
To mitigate the impact of imperfect detection results and integrate pairwise spatial information, we propose spatial-aware pooling (SAP). For simplicity, we omit the upperscript of the human-object pair. Specifically, as shown in Fig. 2 (b), given bounding boxes Bn, B, of a human-object pair, SAP takes the pooled human and object features Jh, So as the input. Then these features are merged through a Multilayer Perceptron (MLP) to formulate the interaction features finter- A cross attention layer aggregates useful information from the image features into finer- Similar Zhaag et al. (2022a), assume B, = [Ir. yh. Wh, ha), where sh and yh denote the center of the bounding box while W/ and hy are the width and height, we leverage the box areas, aspect ratios, intersection over union (IoU), and direction from human to the object to form the pairwise spatial vector,
=
JoU(Bn, Bo), [wpphn wahor by ho
Wh W.
Ih No
Yh Yo
wh
hnp
Espatial = (U)
where & sppatial (U) is MLP to project the spatial vector into the space of finsr- The final MLP merges spatial information in the spatial features into the interaction features.
Cross attention operation between the interaction features and image features allows the information aggregation beyond the area of bounding box, relieving the impact of imperfect detection results as shown in Fig. 5] The output finter of spatial-aware pooling contains both appearance and spatial information of given human-object pair, making it suitable for IR. The new interaction features are used to replace <./mer>> in the question prompt through projection layer.
5

---

Published as a conference paper at ICLR 2026
Our method associates every human instance with every object instance, which leads to large number of non-in non-interactivv human-object pairs. Filtering out non-interactive human-object pairs be- fore sending them into the LLM can significantly reduce the computational complexity. Therefore, based on the interaction features, we add a one linear classifier to predict the interactiveness of a human-object pair,
Simeenetivenness = ((Linear((fmer))
where Sinteractiveness [0, 1] scalar that measures whether a person and an object are interacting, the sigmoid function.
3.4 ONE-PASS DETERMINISTIC MATCHING
With Eq. 5 most non-interactive human-object pairs can be filtered out. However, for each human-object pair, multiple forward passes of the LLM are still needed to calculate the interaction scores for all candidate interactions with Eq. B To further reduce the computational complexity, propose one-pass deterministic matching method to change the deterministic generation feature matching task. Specifically, for a human-object pair B B% C.), and its corresponding candidate interaction list ((C.) = {Ti, T2,..... Tw. J. we add special token < hoi after each candidate interaction, and change the question prompt
The interaction features are <fime>>> Select the correct interaction from the list: T |hoi| >, T |hoi] ....., TM< [hoi] >
Question: <fing
After tokenizing the question prompt Q and replacing the placeholders <fimg>, < with corre- sponding image features and interaction features, respectively, we send the prompt features into the LLM for feature extraction. Assuming the output features of each special token are fhoi [k], where the index of the candidate interaction in the list, the output interaction features finerr the cosine similarity between these two features is used as the confidence score,
[k], ft f k 1,2..... M. [t[k] cosine (fa
(6)
With Eq. 6. for each human-object pair, forward pass of the LLM through features matching.
predict all candidate interactions with only one
3.5 TRAINING AND INFERENCE
Training. train our method with two stages. At the first stage, we train SAP only. Specifically, given set of associated human-object pairs, assign positive label for a human-object pair if the bounding boxes of both human and object have an loU exceeding a threshold with the ground truth. The binary focal loss (Lin al. 2017) is adopted,
Nua
Cinteractiveness
FocalBCB St interactiveness!
Yí
interactivenevss
N Ninter
where Yi internetiveness {0, is the interactiveness label.
At the second stage, we freeze SAP, and train the LLM with LoRA Hu et al. (2022). Specifically, by assigning the human-object pair with a ground truth interaction pair, the labels are determined by the order the positive interaction the candidate interaction list of the question prompt. For example, if the interactions of the ground truth interaction pair are feeding a bird, holding bird, the candidate interaction list in the question prompt is feeding a bird, chasing a bird, holding a bird, the interaction labels are Y (1,0, 1). The binary focal loss is adopted,
Nuva (0(C))
Cmatching
(oca)
(8)
Nintr (CC))
where & is the k-th interaction in the candidate interaction list ((C.), je(C)) denotes the number of interaction the list. (t(k] is obtained with Eq. 6 and clamped into (0, 1) to avoid negative value. Note that, the visual encoder in the MILM is frozen in both training stages.

---

Published as a conference paper at ICLR 2026
Table 1: Performance comparison on HICO-DET under zero-shot, cross-detector and training- free settings. o indicates BLIP2 (Li et al. (2023) features is used. f and indicate Grounding-DING (Liu et al, 2024) and Yolo- World (Cheng et al. 2024) are used as the detector, respectively. is the training-free setting.
RF-UC
NF-UC
UO
UV
Arg Full
Methods
Unseen
Seen
Full
Unseen
Seen
Full
Unseen
Seen
Full
Unseen
Seen
Full
ADA-CM
27.63
33.01
32.41
31.39
ADA-CM
20.82
30.12
28.26
26.23
27.03
26.87
ADA-CM;
20.43
29.23
27.47
24.89
27.69
27.13
BCOM
28.52
35.04
33.74
33.12
31.76
32.03
BCOM
9.23
23.08
20.31
16.38
20.73
19.86
BCOMT
17.69
13.76
17.00
GEN-VLKT
21.36
32.91
30,56
25.05
23.38
23.71
10.51
28.92
25.63
20.96
30.23
28.74
27.16
HOICLIP
25.53
34.85
32.99
26.39
28.10
27.75
16.20
30.99
28.53
24.30
32.19
31.09
30.09
CLIPHHO
28.47
35.48
34,08
31.44
28.26
28.90
31.79
32.73
32.58
26.02
31.14
30.42
31.50
CMMP
29.45
32.87
32.18
32.09
29.71
30.18
33.76
31.15
31.59
26.23
32.75
31.84
31.45
34.27
33.76
EZ-HOI
34.24
37.35
36,73
36,33
34.47
34,84
38.17
36,02
36,38
28.82
38.15
36.84
36.20
UniHOlo
28.68
33.16
32.27
28.45
32.63
31.79
19.72
34.76
31.56
26.05
36.78
34.68
32.08
BC HO-HOL
42.31
40.67
40,99
33.01
37.24
36.40
19.94
37.03
34.18
31.18
41,31
39.89
37.87
Ours
41.79
44.01
43.56
43.12
39.63
40,33
48.67
42.58
43.60
36.89
43.84
42.88
42.59
Ours
43.30
45.19
50.15
45.28
36.88
45.66
44.43
44.00
Ours
43.81
44.05
44,00
43.36
41.67
42.01
47.95
44.19
44.82
36.25
45.13
43.88
43.68
ADA-CM+
25.19
25.19
25.19
25.19
25.19
Ourse
31.50
31.50
31.50
31.50
31.50
Inference. During inference, with the detection results of any detector, we first formulate the human- object pairs, and leverage SAP to filter out the non-interactive pairs whose Sinteractivenees is below threshold X, the remaining human-object pairs are sequentially sent into the MLLM for IR. The final interaction confidence score of the i-th human-object pair is,
[ = S S internctiveness S S k = 1,2
(9)
where S and S are the detection score given by the detector. Our method fully decouples the object detection and IR, and can combine with any detector without retraining.
4 EXPERIMENTS
4.1 EXPERIMENTAL SETTINGS
Datasets. We evaluate our method on two widely used HOI detection benchmarks: HICO- DET (Chao et al.) 2018) and V-COCO (Gupta & Malik 2013). HICO-DET (Chao et al. 2018) contains 47,776 images annotated with 600 HOI categories, defined over 80 object categories and 117 verbs. The default setting splits the dataset with 38,118 images for training and 9,658 for testing. V-COCO Guppta & Malik 2013) is built upon MS COCO and provides annotations for 29 action categories across 10,396 images. We adopt the official train/val/test splits.
Data Structure and Evaluation Metries for Zero-Shot, Cross-Detector and Cross-Dataset Set- tings. To evaluate generalization unseen HOI categories, follow the settings introduced in prior works (Bansal et al. 2020) Hou et al., 2020) on HICO-DET: Rare First Unseen Combination (RF-UC): unseen HOIs are constructed by holding out rare verb-object pairs during training. Non Rare First Unseen Combination (NF-UC): unseen HOIs are constructed by holding out non-rare verb-object pairs. Unseen Verb (UV): unseen HOIs involve verbs never observed during training. Unseen Object (UO): unseen HOIs involve objects never observed during training. To validate the decouple framework, the cross-detector setting uses the detection results with various detectors with- out retraining the model. In addition, we consider a cross-datase setting, training HICO-DET and testing on V-COCO, which poses a more challenging generalization scenario. We report mean Average Precision (mAP) an evaluation metric. An HOI triplet is considered correct if the human and object boxes achieve loU 2 0.5 with ground truth and the interaction category is correct.
Implementation Details. For the object detector, we adopt ResNet50 DETR (Carion et al. 2020 Grounding-DDie DINO (Liu et al. 2024). and YoLo-World (Cheng et al., 2024). The MLLM is Qwen 2.5-VL 3B (Bai al. 2025). Training-free free inference uses our proposed deterministic generation method. Fine-tuning is conducted with Adam W (Loshchilov & Hutter 2017). ). At the first stage, we train SAP for 30 epochs with learning rate of 0.0001 and batch size of 16. At the second stage,
7

---

Published as a conference paper at ICLR 2026
only fine-tune the LoRA in the LLM for 16 epochs with learning rate of 0.0001 and batch size of 16. We use a similar data augmentation with most of HOI detection methods (Zhagg et al. 2022a; 2023 Ning et al. 2023). During training, use the ground truth bounding box of objects as the detection results. During inference, the human-object pairs whose interaction score below 0.15 is filtered out. All experiments conducted on 4 Nvidia RTX 3090s.
4.2 COMPARISON WITH OTHER METHODS
Zero-shot Settings. Table 1 summarizes the comparison with other methods on HICO-DET under four zero-shot settings. The comparison methods include one-stage methods: GEN-VLKT (Liao et al.] 2022), HOICLIP (Ning et al.) 2023), two-stage methods: ADA-CM (Lei et al.) (023),
CLIP4HOI Mao et al. 2024), BCOM (Wang et al. (024), CMMP (Lei et al., 2025), LAIN (Kim)
ct al. 2025 and methods that leverage BLIP2 (Li et al. 2023) or MLLM: UniHOI (Cao et al.
2023 BOIHOI (Hu et al. 2025), EZ-HOI (Lei et al. (024a). Our method consistently achieves the best performance across all settings. Its generalization ability unseen interactions is impressive,
e.g. it surpasses BC-HOI by about 10. 1 11 and 28.73% on the NF-UC and UO settings.
Cross-Detector Setting. Once trained, our framework allows the detector be freely changed with- out retraining on HICO-DET. When combined with Grounding-DDin DINO or Yolo- Yolo-World, our method further improves the average mAP to 44.00% and 43.68%. This detector-agnostic property high- lights the modularity of our method: stronger object detectors can directly boost HOI detection per- formance. In contrast, most methods rely on the features from specific detector. Only ADA-CM and BCOM do not rely on detector features. However, they still cannot change the detectors without retraining, e.g., replacing the original ResNet-50 DETR with other detectors (Grounding-Diie DINO or Yolo-World) leads to degraded performance. BCOM leverages UPT that models the relationship be- tween different detected objects, while ADA-CM introduces an instance-awar adapter that uses the prior knowledge of detected objects. With these designs, the model training relies on the detection results of a specific detector, and inter-object relations, leading to the coupling of the detector and the interaction recognizer.
Table 2: Performance comparison on V-COCO under cross-dataset setting.
Training-Free Setting. Our method achieves 31.50% mAP, outperforming ADA-CM by clear margin. Even without any training, our method achieves performance comparable methods that rely complex training strat- egy, e.g., UniHOI, CMMP.
V-COCO
Methods
mAp#2 rale
GEN-VLKT
42.34
ADA-CM
46.39
Cross-Datasel Setting. To further validate the general- ization of our method, we conduct a more challenging cross-dataset setting, where the model is trained on HICO- DET and evaluated on V-COCO. As shown in Table P our method achieves a substantial improvement over prior works, reaching 59.91% % mAp#2 rale: These promising results demonstrate the strong generalization of
HOICLIP
41.82
BCOM
48.87
CMMP
47.65
LAIN
48.34
Ours
59,91
leveraging MLLMs for IR, while the detector-agnostic design makes our method both highly flexible and broadly applicable.
4.3 ABLATION STUDY
Effect of Deterministic Generation. To evaluate the effect of deterministic generation, we con- duct experiments under the training-free setting with different formulations of question prompts in Table 3 "Simple" is the simple QA format. "Multiple Choice Questions" denotes formulating the candidate interactions as multiple choices and the model should give the correct choices. "In Con- text" denotes some example QA pairs are given in the prompt. The open-ended text generation pro- cedure of the MILM leads to high format error rate. More importantly, the model has a strong bias towards single-interaction predictions, leading to low interaction recognition performance. Lever- aging conditional likelihood as confidence score brings limited improvement. In contrast, our outputs,
proposed deterministic generation eliminates format errors and ensures boosting the mAP to 31.50%. Moreover, deterministic generation is also important to the model af- ter supervised mhe-tunnng (SFT), improving the performance from 31.61% to 39.87%. These results demonstrate that deterministic generation is essential for both training- free and trained models.
8

---

Published as a conference paper at ICLR 2026
Table 3: Ablation studies on the deterministic generation. "Simple", "Multiple Choice Questions" "In Context" are the different formulations of question prompts. Some examples shown in A.5 "*" denotes leveraging the conditional likelihood of the answer as the confidence score of the pre- diction. "Format Error Rate" denotes the rate which the format of the answer is error. "Single Output Rate" indicates the rate at which the answer only contains one interaction. "SFT" indicates the model fine-tuned the HICO-DET with supervised fine-tuning.
Settings
Full
Format Error Rate
Single Oulput Rate
Simple
14.23
36.78
80.91
Multiple Choice Questions
18.91
23.75
80.17
In Context
19.57
18.75
78.82
Simples
16.19
36.78
80.91
Multiple Choice Questions*
20.36
23.75
80.17
In Context*
20.97
18.75
78.82
Deterministic Generation
31,50
0
SFT
31.61
6.27
58.23
SFT + Deterministic Generation
39.87
0
Table 4: Ablation studies each proposed component. The model trained with SFT and infer- ence with deterministic generation serves as the baseline. "SAP" and "DM" are the spatial-awan pooling module and deterministic matching, respectively. "Pairwise Spatial" and "Cross Attention" are the integration of pairwise spatial information and cross attention to aggregate image features, respectively. "UPT" indicates the unary- saary-pairwise transformer is used instead of SAP.
Inference
Methods
Unseen
Seen
Full
Unseen
Seen
Full
Time (ms)
Baseline
43.66
38.36
39.24
34.06
38.46
37.84
569
w/ SAP DM
45.91
41.59
42.31
36.47
42.84
41.95
217
45.63
39.47
40.50
36.14
39.75
39.24
189
w/o Pairwise Spatial
44.31
41.52
41.98
35.32
41.66
40.77
w/o Cross Attention
43.61
40.92
41.37
34.50
41.76
40.74
87
UPT
43.66
41.38
41.76
35.29
41.44
40.58
122
Ours
48.67
42.58
43.60
36.89
43.84
42.88
Effect of Each Proposed Component. Table 4 annalyess the contributions of spatial-aware pooling (SAP) and deterministic matching (DM). The inference time denotes the average inference time per image. With SAP, most of non-interaction human-object pairs are filtered out. DM further enables the prediction of multiple candidate interactions in single forward pass. These two components bring clear improvements over the baseline and reduce the computational complexity.
We also evaluate the design of SAP. Removing either pairwise spatial encoding or cross attention results in noticeable drops in both UO and UV settings. Specifically, discarding spatial encoding reduces the ability to reason about relative human-object positions, while discarding cross attention weakens the robustness noise in the detection results. The unary-pairwisg transformer (Zhang) et al. 2022a) is widely used by many methods (Lei et al.) 2025) Mao et al.) 2024) to integrate the pairwise spatial and appearance information. We replace SAP with UPT, and change its input with the human and object features obtained by ROIAlign. This setting leads performance drop and a coupling of the detector as UPT leverages the relationship between different detected objects.
To investigate the effect of candidate ordering in the question prompt, we perform inference times using different permutations the candidate list. As shown in Table method remains robust to different candidate orders, with only minor performance fluctuations.
Impact of Candidate Order.
Table 5: Impact of the Candidate Order.
Unseen
Seen
Full
48.65 0.03
42.59 t 0.02
43.61 = 0.02

---

Published as a conference paper at ICLR 2026
5 CONCLUSION
This paper proposes decoupled framework for zero-shot HOI detection, which separates object detection from interaction recognition (IR), and exploits the powerful multi-modal large language models (MLLMs) for IR. By casting IR as VQA task, method achieves superior performance in the training-free setting with deterministic generation. To further overcome the bottlenecks of MLLMs for IR, we introduce spatial-aware pooling module that integrates pairwise spatial infor- mation, and one-pass deterministic matching method that reduces the computational complexity by converting the generation feature matching task. Extensive experiments on HICO-DET and V-coCo demonstrate the effectiveness of our method, which achieves impressive performance un- der zero-shot and cross-dataset settings. Once trained, our method has the flexibility to integrate with any object detector without retraining and can benefit from the progress of detectors.
ACKNOWLEDGEMENT
This work supported in part by Natural Science Foundation of China under Grant No. 62425603, 62506167, 62500397, in part by the Basic Research Program of Jiangsu Province under Grant No. BK2024001 BK20251451, in part by the Fundamental Research Funds for the Central Universities under Grant No. 30925010206, in part by Natural Science Foundation of Sichuan Province under Grant No. 2026NSFSC1449.
REFERENCES
Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, al. Qwen2. 5-vl technical report. arXiv preprint arXiv:2502. 13923, 2025.
Ankan Bansal, Sai Saketh Rambhatla, Abhinav Shrivastava, and Rama Chellappa. Detecting human- object interactions via functional generalization. In AAAI, volume 34, pp. 10460-10469, 2020.
Yichao Cao, Qingfei Tang, Xiu Su, Song Chen, Shan You, Xiaobo Lu, and Chang Xu. Detecting any human-object interaction relationship: Universal hoi detector with spatial prompt learning foundation models. In NeurIPS, volume 36, pp. 739-751, 2023.
Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In ECCV, pp. 213-229. Springer, 2020.
Yu-Wei Chao, Yunfan Liu, Xieyang Liu, Huayi Zeng. and Jia Deng. Learning detect human-object interactions. In WACV, pp. 381 381-389. IEEÉ, 2018
Mingfei Chen, Yue Liao, Si Liu, Zhiyuan Chen, Fei Wang, and Chen Qian. Reformulating hoi 2021.
detection as adaptive set prediction. In CVPR, pp.
Tianheng Cheng, Lin Song, Yixiao Ge, Wenyu Liu, Xinggang Wang, and Ying Shan. Yolo-world: Real-time open-vocabulary object detection. In CVPR, pp. 16901-16911, 2024.
Leizhen Dong, Zhimin Li, Kunlun Xu, Zhijun Zhang, Luxin Yan, Sheng Zhong, and Xu Zou. Category-aware transformer network for better human-object interaction detection. In CVPR, pp. 19538-19547, June 2022.
Qi Dong, Zhuowen Tu, Haofu Liao, Yuting Zhang, Vijay Mahadevan, and Stefano Soatto. Visual relationship detection using part-and-sum transformers with composite queries. 3550-3559, 2021.
ICCV, pp.
Chen Gao, Yuliang Zou, and Jia-Bin Huang. ican: Instance-centric attention network for human- object interaction detection. arXiv preprint arXiv:1808.1 10437, 2018.
Chen Gao. Jiarui Xu, Yuliang Zou, and Jia-Bin Huang. Drg: Dual relation graph for human-object interaction detection. In ECCV, pp. 696-712. Springer, 2020.
10

---

Published as a conference paper at ICLR 2026
Georgia Gkioxari, Ross Girshick, Piotr Dollár, and Kaiming He. Detecting and recognizing human- object interactions. In CVPR, June 2018.
Saurabh Gupta and Jitendra Malik. Visual semantic role labeling. arXiv preprint ar..v::505.044:. 2015.
Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-enn. In ICCV, pp. 2961- 2017.
Zhi Hou, Xiaojiang Peng, Yu Qiao, and Dacheng Tao. Visual compositional learning for human- object interaction detection. In ECCV, pp. 584 600. Springer, 2020.
Zhi Hou, Baosheng Yu, Yu Qiao, Xiaojiang Peng, and Dacheng Tao. Detecting human- buman-object inter- action via fabricated compositional learning. In CVPR, pp- 14646-14655, 2021.
Edward Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models. ICLR, 12):3, 2022.
Yupeng Hu, Changxing Ding, Chang Sun, Shaoli Huang, and Xiangmin Xu. Bilateral collaboration with large lision-language models for open vocabulary human-object interaction detection. arXiv preprint arXiv:2507.063:
2025.
Yixiang Jin, Dingzhe Li, Jun Shi, Peng Hao, Fuchun Sun, Jianwei Zhang, Bin Fang, al. Robotgpt: Rohot manipulation learning from chatgpt. IEEE Robotics and Automation Letters, 9(3):2543- 2550, 2024.
Bumsoo Kim, Junhyun Lee, Jaewoo Kang, Eun-Sol Kim, and Hyunwoo J Kim. Hotr: End-to-end human-object interaction detection with transformers. In CVPR, pp. 74-83, 2021.
Bumsoo Kim, Jonghwan Mun, Kyoung- Woon On, Minchul Shin, Junhyun Lee, and Eun-Sol Kim. Mstr: Multi-scale transformer for end-to-end human-object interaction detection. In CVPR, pp- 19578-19587 2022.
Sanghyun Kim, Deunsol Jung, and Minsu Cho. Locality-awand zero-shot human-object interaction detection. In CVPR, pp. 20190-20200, June 2025.
Qingian Lei, Bo Wang, and Robby T. Tan. Ez-hoi: Vlm adaptation via guided prompt learning for zero-shot hoi detection. In NeurIPS, volume 37, pp. 5583 55831-55857, 2024a.
Ting Lei, Fabian Caba, Qingchao Chen, Hailin Jin, Yuxin Peng. and Yang Liu. Efficient adaptive human-object interaction detection with concept- memory. In ICCV, pp.
2023.
Ting Lei, Shaofeng Yin, and Yang Liu. Exploring the potential of large foundation models for open-vocabulary hoi detection. In CVPR, pp. 16657-16667, June 2024b.
Ting Lei, Shaofeng Yin, Yuxin Peng, and Yang Liu. Exploring conditional multi-modal prompts for zero-shot hoi detection. In ECCV, pp. 1-19. Springer, 2025.
Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Peiyuan Zhang, Yanwei Li, Ziwei Liu, al. -lavvoooevisoon: Easy visual task transfer. arXiv preprint arXiv:2408.0 03326, 2024a.
Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. BLIP-2: Bootstrapping inguaage-maage pre-training with frozen image encoders and large language models. In ICML, volume 202, pp. 19730-19742. PMLR, 23-29 Jul 2023.
Zhuolong Li, Xingao Li, Changxing Ding, and Xiangmin Xu. Disentangled pre-training for human- object interaction detection. In CVPR, pp. 28191 -2201,, June 2024b.
Bencheng Liao. Shaoyu Chen, Haoran Yin, Bo Jiang, Cheng Wang, Sixu Yan, Xinbang Zhang, Xiangyu Li. Ying Zhang, Qian Zhang, et al. Diffusiondrive: Truncated diffusion model for end- to-end autonomous driving. In CVPR, pp. 12037-120477 2025.
2025.

---

Published as a conference paper at ICLR 2026
Yue Liao, Si Liu, Fei Wang, Yanjie Chen, Chen Qian, and Jiashi Feng. Ppdm: Parallel point de- tection and matching for real-time human-objec, interaction detection. In CVPR, pp. 482-490 2020.
Yue Liao, Aixi Zhang, Miao Lu, Yongliang Wang, Xiaobo Li, and Si Liu. Gen-vikt: Simplify asso- ciation and enhance interaction understanding for hoi detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 20123-20132, 2022.
Tsung-Y Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár. Focal loss for dense object detection. In ICCV, pp. 2980-2988, 2017.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. NeurIPS, 36:34892-34916. 2023.
Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang. Jie Yang, Qing Jiang, Chunyuan Li, Jianwei Yang. Hang Su, et al. Grounding dino: Marrying dino with grounded pre-training for open-set object detection. In ECCV, pp. 38-55. Springer, 2024.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint arXiv:1711.0501,, 2017.
Yunyao Mao, Jiajun Deng, Wengang Zhou, Li Li, Yao Fang, and Houqiang Li. Clip4hoi: Towards adapting clip for practical zero-shot hoi detection. NeurIPS, 36, 2024.
Gyeongsik Moon, Heeseung Kwon, Kyoung Mu Lee, and Minsu Cho. Integralaction: Pose-driven feature integration for robust human action recognition in videos. In CVPR. pp. 3339-3348, 2021.
Shan Ning, Longtian Qiu, Yongfei Liu, and Xuming He. Hoiclip: Efficient knowledge transfer for hoi detection with vision-language models. In CVPR, pp. 23507-23517, 2023.
Siyuan Qi, Wenguan Wang, Baoxiong Jia, Jianbing Shen, and Song-Chun Zhu. Learning human- object interactions by graph parsing neural networks. In ECCV, pp. 401 2011-11,220 2018.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In ICML, pp. 8748 8763. PMLR, 2021.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer: High- resolution image synthesis with latent diffusion models. In CVPR, pp. 10684-10695, 2022.
Mohit Shridhar, Lucas Manuelli, and Dieter Fox. Cliport: What and where pathways for robotic manipulation. In Conference on robot learning. pp. 894-906. PMLR, 2022.
Masato Tamura, Hiroki Ohashi, and Tomoaki Yoshinaga. Qpie: Query-based pairwise human-object interaction detection with image-wide contextual information. In CVPR, pp.
2021.
Danyang Tu, Xiongkuo Min, Huiyu Duan, Guodong Guo, Guangtao Zhai, and Wei Shen. Iwin: Human-object interaction detection via transformer with irregular windows. In ECCV, pp. 87- 103. Springer, 2022.
Oytun Ulutan, ASM Iftekhar, and Bangalore S Manjunath. Vsgnet: Spatial attention network for detecting human object interactions using graph convolutions. CVPR, pp. 13617-13626, 2020.
Bo Wan, Desen Zhou, Yongfei Liu, Rongjie Li, and Xuming He. Pose-aware multi-level feature network for human object interaction detection. In ICCV, October 2019.
Guangzhi Wang, Yangyang Guo, Ziwei Xu, and Mohan Kankanhalli. Bilateral adaptation for homan-object interaction detection with occlusion-robuustess. CVPR, pp. 27970-27980, 2024.
Jingbo Wang, Ye Yuan, Zhengyi Luo, Kevin Xie, Dahua Lin, Umar Iqbal, Sanja Fidler, and Samch Khamis. Learning human dynamics in autonomous driving scenarios. In ICCV, pp. 20796-20806, 2023.
Tiancai Wang, Tong Yang. Martin Danelljan, Fahad Shahbaz Khan, Xiangyu Zhang, and Jian Sun. Learning human-object interaction detection using interaction points. In CVPR, June 2020.
12

---

Published as a conference paper at ICLR 2026
Mingrui Wu, Xuying Zhang, Xiaoshuai Sun, Yiyi Zhou, Chao Chen, Jiaxin Gu, Xing Sun, and Rongrong Ji. Difnet: Boosting visual information flow for image captioning. CVPR, pp. 18020-18029, June 2022.
Ting Yao, Yingwei Pan, Yehao Li, and Tao Mei. Exploring visual relationship for image captioning. In ECCV, pp. 684-699, 2018.
Hangjie Yuan, Jianwen Jiang, Samuel Albanie, Tao Feng, Ziyuan Huang, Dong Ni, and Minggian Tang. Rlip: Relational language-image pre-training for human-object interaction detection. NeurIPS, 35:37416-37433 2022.
Hangjie Yuan, Shiwei Zhang, Xiang Wang, Samuel Albanie, Yining Pan, Tao Feng. Jianwen Jiang, Dong Ni, Yingya Zhang, and Deli Zhao. Rlipv2: Fast scaling of relational language-image pre- training. In ICCV, pp. 21649-216660 2023.
Frederic Z Zhang, Dylan Campbell, and Stephen Gould. Spatially conditioned graphs for detecting human-object interactions. In CVPR, pp. 13319-13327, 2021.
Frederic 7. Zhang, Dylan Camphell, and Stephen Gould. Efficient two-stage detection of human- object interactions with novel unary-pairwise transformer. In CVPR, pp. 20104-201 12, 2022a.
Frederic Z Zhang, Yuhui Yuan, Dylan Campbell, Zhuoyao Zhong, and Stephen Gould. Exploring predicate visual context in detecting of human- interactions. In ICCV, pp. 10411-10421, 2023.
Yong Zhang, Yingwei Pan, Ting Yao, Rui Huang, Tao Mei, and Chang-Wen Chen. Exploring structure-awar transformer over interaction proposals for human-object interaction detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp- 19548-19557, 2022b.
Desen Zhou, Zhichao Liu, Jian Wang, Leshan Wang, Tao Hu, Errui Ding, and Jingdong Wang. Human-objeet interaction detection via disentangled transformer. In CVPR, pp. 19568-19577, 2022.
Penghao Zhou and Mingmin Chi. Relation parsing neural network for human-object interaction detection. In ICCV, October 2019.
Cheng Zou, Bohan Wang, Yue Hu, Junqi Liu, Qian Wu, Yu Zhao, Boxun Li, Chenguang Zhang, Chi Zhang, Yichen Wei, et al. End-to-end human object interaction detection with hoi transformer. In CVPR, pp. 11825-1 1834, 2021.
13


Reviews:
[{"review_id": "dJtkOFsLlt", "reviewer": "Reviewer_fri8", "summary": "This paper propose a decoupled framework that separates object detection from IR and leverages multi-modal large language models (MLLMs) for training-free zero-shot IR, with a spatial-aware pooling module and a one-pass deterministic matching method to enhance performance and efficiency. Extensive experiments demonstrate that this method achieves superior zero-shot performance, strong cross-dataset generalization, and the flexibility to integrate with any object detectors without retraining.", "questions": "See limitations and cons above.", "limitations": "", "rating": 4, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. Clear Motivation. The paper identifies the key limitations of existing HOI approaches, including coupling with a specific detector, poor generalization, and coarse-grained VLM features. In response, This paper propose a method that explicitly decouples object detection from interaction recognition, supports flexible plug-and-play integration with any detector, and demonstrates strong generalization capability.\n\n2. In contrast to prior work that relies on CLIP embeddings, this paper introduces a novel deterministic generation strategy to address application bottlenecks of MLLMs in zero-shot HOI tasks. The proposed framework further implements one-pass deterministic matching, which substantially improves computational efficiency.\n\n3. Experiments show that the proposed method surpasses other baselines under zero-shot setting, training-free setting and cross-dataset setting. Ablation studies confirm the effectiveness of each individual component in the framework.", "weaknesses": "1. Limited Novelty. The proposed method appears relatively simple and heavily relys on the capabilities of open-vocabulary detectors and MLLMs, which also means it inherits their inherent weaknesses, such as missed detections, redundant detections, incorrect category predictions in detectors, and hallucination issues in MLLMs.\n\n2. The method first defines the candidate interaction list based on the categories of detected objects, thereby reformulating the HOI task into a multi-label classification problem for the MLLM. This design may considerably restrict the applicability of the model in open-world scenarios.\n\n3. While the two datasets are widely used, they are relatively outdated(2015 & 2018). It remains unclear whether the approach can handle certain complex cases (e.g., multi-object, dense crowds situations) or perform robust open interaction recognition. It would be valuable to include experiments or visualizations demonstrating performance on new tasks mentioned in the introduction, such as robotic manipulation or autonomous driving.\n\n4. Experimental results on the SWiG-HOI dataset are absent.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "5LXAp2alUI", "reviewer": "Reviewer_jVQb", "summary": "This paper introduces a novel framework for zero-shot Human-Object Interaction (HOI) detection that fully decouples object detection from interaction recognition. It leverages Multi-modal Large Language Models (MLLMs) by formulating interaction recognition as a Visual Question Answering task, enabled by a deterministic generation method for training-free zero-shot inference. To further boost performance and efficiency, the authors propose a Spatial-Aware Pooling module to integrate appearance and spatial cues, and a One-Pass Deterministic Matching strategy that converts generation into efficient feature matching. Extensive experiments demonstrate state-of-the-art performance under various zero-shot settings and strong cross-dataset generalization.", "questions": "Please kindly refer to the weakness section.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- This paper investigates an interesting question of how to utilize MLLM to perform the HOI detection task.\n- The presentation is clear and easy to follow.\n- The proposed One-Pass Deterministic Matching is effective and interesting.", "weaknesses": "-  Lack of Justification for the One-Pass Prompt Design. The proposed one-pass method, which appends the same special token 
 after each candidate interaction in the prompt (Line 286), requires further theoretical and empirical justification. This design raises a fundamental concern regarding the contextual representation of the 
 tokens within the Transformer architecture. Specifically, the representation of a particular 
 token (e.g., the one following candidate T_n) is computed through self-attention over the causal context—it can attend to all preceding tokens, including the text of T_1 to T_n and all previous 
 tokens, but it is inherently blind to the subsequent candidates (T_{n+1} to T_M).\n- Potential Instability Due to Candidate Order Sensitivity. Following the above question, the asymmetric contextual representation leads to a critical, unexplored question: Would the model's prediction for a given candidate interaction be sensitive to the order in which the candidates are listed? If the candidate list is permuted, the contextual information available to each 
 token changes, which may lead to inconsistent similarity scores and final predictions. Such order sensitivity would be a significant drawback for a robust system, as the candidate set is inherently unordered. The authors must empirically evaluate this potential instability, for instance, by measuring performance variance under different random permutations of the candidate list.\n- Missing some efficiency comparison with the traditional HOI methods. The ablation study on inference time (Table 4) convincingly shows the efficiency gains from the proposed components. However, to better position the work, a comparison of the overall inference speed against other state-of-the-art HOI methods is necessary, such as BC-HOI and EZ-HOI. Furthermore, please specify the hardware configuration and batch size used for the timing measurements.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "8lR59eQ6Um", "reviewer": "Reviewer_4117", "summary": "The paper proposes a decoupled HOI pipeline that keeps object detection separate from interaction recognition and uses an MLLM to classify interactions.\nKey ingredients are a “deterministic generation” proxy for multi‑label IR, a spatial‑aware pooling (SAP) module that fuses ROI features with pairwise spatial cues via cross‑attention, and a one‑pass deterministic matching scheme to score all candidate verb–object pairs in a single forward pass.\nThe method reports gains on HICO‑DET zero‑shot splits, a cross‑detector setting (e.g., swapping in Grounding‑DINO or YOLO‑World), and cross‑dataset transfer to V‑COCO, with extensive ablations.", "questions": "- What exactly is included in the latency reported in Table 4? Please report end‑to‑end wall‑clock time, memory, and GPU count for detection + pairing + IR at test time, and compare to one‑stage methods at similar accuracy.\n\n- Can the authors report results when the MLLM directly predicts HOI triplets in an open-ended manner (i.e., free-form verb–object labels per human–object pair, without the candidate list?", "limitations": "", "rating": 4, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- The approach yields strong zero‑shot results on HICO‑DET, including a training‑free variant (31.5 mAP), and improves further with LoRA fine‑tuning and the proposed modules; it also transfers well to V‑COCO (59.91 mAP).\n\n- The paper demonstrates plug‑and‑play use with multiple detectors (ResNet‑50 DETR, Grounding‑DINO, YOLO‑World) without retraining the IR head. This supports the “detector‑agnostic” design goal.", "weaknesses": "- From both architectural and modular perspectives, the proposed method exhibits limited novelty. Architecturally, it essentially follows the conventional two-stage HOI detection paradigm, where the detection and interaction recognition stages are sequentially processed. The only modification lies in substituting these two components with more powerful counterparts, an open-vocabulary detector (e.g., Grounding-DINO) and an MLLM-based interaction recognizer, without introducing new mechanisms or architectural restructuring. At the module level, each component is constructed from widely used elements: SAP integrates ROIAlign, MLPs, and cross-attention layers, coupled with a standard pairwise spatial vector; interactiveness is estimated via a simple linear classifier; and the deterministic generation/matching merely reformulates scoring through prompt-based likelihood estimation and LoRA fine-tuning, which are established techniques in existing MLLM literature. Consequently, the methodological advancement is incremental relative to prior two-stage and CLIP/MLLM-augmented HOI frameworks\n\n- The claimed zero‑shot setting is substantially aided by powerful pretrained components whose training corpora likely cover many HOI objects and patterns. Implementation explicitly relies on Grounding‑DINO/YOLO‑World for detection and Qwen2.5‑VL for IR, and the candidate interaction list is built from object categories, meaning much of the supervision is imported from large‑scale pretraining and the dataset’s curated verb–object space. As a result, the evaluation is closer to dataset‑level zero‑shot composition under heavy external priors than to strict zero‑data transfer.\n\n- Although one‑pass matching and interactiveness filtering reduce the number of MLLM calls, the pipeline still couples a heavy open‑vocabulary detector with an MLLM‑based pairwise scorer, and the number of pairs scales quadratically with detections. The reported per‑image times in Table 4 appear to measure the IR component; it is unclear whether detector runtime, prompt construction, and tokenization are included, and how the overall throughput compares to strong one‑stage baselines at matched accuracy.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "W71FBUMTkr", "reviewer": "Reviewer_zR4B", "summary": "This paper leverages Multi-modal Large Language Models to address HOI detection problem and decoupled object detection and interaction recognition. Interaction recognition is reformulated as a VQA task to leverage the MLLM abilities. The authors propose two pipelines, a training-free deterministic generation method, which replaces open-ended text generation with conditional likelihood estimation; and a fine-tuning pipeline, which enhances efficiency with a spatial-aware pooling module and a one-pass deterministic matching strategy. The method achieves strong zero-shot and cross-dataset performances.", "questions": "1. I have a question about the Table 2 for cross-dataset evaluation. Since V-COCO and HICO-DET have different pre-defined HOI classes, only baselines supporting to add new HOI classes in inference can be evaluated in this cross-dataset evaluation. Then, I wonder how ADA-CM is evaluated in table 2, whose fine-tuning version seems to be unable to add new classes in inference.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper first leverages MLLM to solve HOI problem, which is novel and promising, as it connects high-level vision-language reasoning with structured HOI understanding.\n2. The paper presents two pipelines, a training-free deterministic pipeline and a fine-tuned version and both achieve competitive performances.\n3. Traditional MLLM-based generation is computationally expensive and unstable for classification-style tasks. The paper proposes to reformulate the open-ended generation into a deterministic generation problem. In fine-tuning version, it proposes a single-pass feature matching task using HOI tokens and cosine similarity, improving efficiency.\n4. The experiments systematically compare MLLMs of different scales, demonstrating consistent improvements as model size increases (from 0.5B to 7B). This analysis provides valuable evidence that scaling multimodal models enhances HOI reasoning, further validating the proposed method.", "weaknesses": "1. Since the method leverages the MLLMs, naturally with strong open-vocabulary capability evaluating on open-vocabulary benchmarks such as SWiG-HOI [a] would better demonstrate its true generalization to unseen categories.\n2. The paper lacks failure case analysis or qualitative discussion. Despite leveraging a large multimodal model, the best mAP remains around 45 in Table 1, 6, 7, suggesting unaddressed limitations.\n3. The paper mainly improves MLLM with spatial information, while HOI detection inherently involves both spatial understanding (detection) and semantic reasoning (interaction recognition). It remains unclear whether original MLLM reasoning is already sufficient for HOI detection. A deeper discussion on this design choice (focusing on spatial information) would strengthen the overall analysis.\n\nReferences:\n\n[a] Learning Transferable Human-Object Interaction Detector with Natural Language Supervision, CVPR2022", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)