InstructHOI: Context-Aware Instruction for Multi-Modal Reasoning in Human-Object Interaction Detection
Authors: Jinguo Luo
Weihong Ren
Quanlong Zheng
Yanhao Zhang
Zhenlong Yuan
Zhiyong Wang
Haonan Lu
Honghai LIU
Venue: NeurIPS 2025 spotlight ()
Keywords: Human-Object Interaction Detection; Interaction Reasoning
Primary area: applications

Abstract: Recently, Large Foundation Models (LFMs), e.g., CLIP and GPT, have significantly advanced the Human-Object Interaction (HOI) detection, due to their superior generalization and transferability. Prior HOI detectors typically employ single- or multi-modal prompts to generate discriminative representations for HOIs from pretrained LFMs. However, such prompt-based approaches focus on transferring HOI-specific knowledge, but unexplore the potential reasoning capabilities of LFMs, which can provide informative context for ambiguous and open-world interaction recognition. In this paper, we propose InstructHOI, a novel method that leverages context-aware instructions to guide multi-modal reasoning for HOI detection. Specifically, to bridge knowledge gap and enhance reasoning abilities, we first perform HOI-domain fine-tuning on a pretrained multi-modal LFM, using a generated dataset with 140K interaction-reasoning image-text pairs. Then, we develop a Context-aware Instruction Generator (CIG) to guide interaction reasoning. Unlike traditional language-only instructions, CIG first mines visual interactive context at the human-object level, which is then fused with linguistic instructions, forming multi-modal reasoning guidance. Furthermore, an Interest Token Selector (ITS) is adopted to adaptively filter image tokens based on context-aware instructions, thereby aligning reasoning process with interaction regions. Extensive experiments on two public benchmarks demonstrate that our proposed method outperforms the state-of-the-art ones, under both supervised and zero-shot settings.

InstructHOI: Context-Aware Instruction for Multi-Modal Reasoning in Human-Object Interaction Detection
Jinguo Luo!.3
Weihong Ren',
Quanlong Zheng",
Yanhao Zhang",
Zhenlong Yuan", Zhiyong Wang', Haonan Lu²,
Honghai Liu²
Harbin Institute of Technology, Shenzhen 3PPO Al Center
Institute of Computing Technology, Chinese Academy of Sciences
(23s153135, weihongren, zhiyongwang, honghai 1iu) @hit edu. ( (zhengquanlong, zhangyanhao, luhaonan) @oppo. com yuanzhenl www.ibbiiitt...........mmm cc.
Abstract
Recently, Large Foundation Models (LFMs), e.g.. CLIP and GPT, have significantly advanced the Silmann-bbjed Interaction (HOI) detection, due to their superior gen- eralization and transferability. Prior HOI detectors typically employ single- or multi-modal prompts generate discriminative representations for HOIs from pretrained LFMs. However, such prompt-based approaches focus on transferring HOI-specific knowledge, but unexplore the potential reasoning capabilities of LFMs, which can provide informative context for ambiguous and open-world interaction recognition. In this paper, propose InstructHOI, novel method that leverages context-aware instructions to guide multi-modal reasoning for HOI detec- tion. Specifically, to bridge knowledge gap and enhance reasoning abilities, we first perform HOI-domain fine-tuning pretrained multi-modal LFM, using gener- ated dataset with 140K interaction-reasone Context-aware Instruction Generator (CIG) to guide interaction reasoning. Unlike traditional language-only instructions, CIG first mines visual interactive context at the human-object level, which is then fused with linguistic instructions, forming multi-modal reasoning guidance. Furthermore, an Interest Token Selector (ITS) is adopted adaptively filter image tokens based context-aware instructions, thereby aligning reasoning process with interaction regions. Extensive experiments two public benchmarks demonstrate that our proposed method outperforms the ttate-oftthe-st ones, under both supervised and zero-shot settings.
image-text pairs. Then, we develop a
Introduction
Human-Object Interaction (HOI) detection plays crucial role in high-level human- under- standing, with applications across various domains 2 3]. The purpose of HOI detection is to detect a series of interactive triplets (i.e.. (human, action. object)) in open-world scenarios. This task can be specifically divided into two sub-tasks: localizing interactive human-object pairs and recognizing their interaction relationships.
Traditional HOI detectors can be primarily classified into one-stage and two-stage approaches. One- stage methods 4
5 6
treat HOI detection unified multi-task learning problem, utilizing
CCorresponding Author Project Leader
39th Conference on Neural Information Processing Systems (NcurIPS 2025)

---

Reasoning Guidance
HOI Spec Knowledge
Ansoon Interaction
Intoraction Thats
Promet
Prefmined LFM
X
Knowledge Transfer
Probubiltty Oulput
Linguat Instnctiion
Interaction
Reasoning
pair2: Hamin is hioding
pair-1:
par-2
LORA
10.0...0.1)
Visual Contect
ahrboard
Interaction Classifler
pal-1
[..0....0)
Interaction Reasoner
Pain-ovel Reasoning Texta from LFM
Imsage
Image
(a) Traditional Knowledge Transfer Methods
(b) Context-Aware Instruction for Interaction Reasoning
Figure 1: (a) Traditional LFM-hased methods typically employ predefined or learnable prompts to transfer HOT-specific knowledge from pretrained LFM for interaction classification. (h) Our InstructHOI integrates visual interactive context with linguistic instructions guide HOI-domain LFM in performing multi-modal interaction reasoning, generating pair-level interaction texts.
multi-branch network simultaneously perform human-object pair detection and interaction prediction. In contrast, two-stage ones [89 first detect human-object instances using an off-the-shelf object detector, subsequently predicting interaction categories based the visual features extracted
from instance areas. Despite significant efforts in feature extraction strategies [10 [T [2] and
architecture improvements [13
[4 15 16), accurately identifying complex HOIs within open- world
context remains challenging when relying solely on visual representation learning.
To further explore discriminative interaction representations, recent researches transfer HOI-specific knowledge from pretrained LFMs, including Vision-Language Models (VLMs) and Large Language Models (LLMs), using predefined or learnable prompts, as illustrated in Fig. Ta). For the VLM- style methods, some earlier works 17
18. 19 leverage static template prompts (e.g.. "a photo of person [action] a/an [object]") to derive linguistic prior knowledge from CLIP (20] at the category level. Subsequently, CMMP [21] introduces learnable multi-modal prompts, facilitating the adaptive transfer of semantic knowledge from CLIP the instance level. Furthermore, the L.M-sttlle adopt language foundation models (e.g., ChatGPT ) to generate finer-grained descriptive texts as interactive clue prompts, thereby transferring the generalizable knowledge of LLMs for HOI detection. However, these prompt-based methods primarily focus on knowledge transfer. but fail to exploit LFMs' reasoning capabilities which can provide informative context for ambiguous and open- open-world interaction recognition.
methods 22 23 24
According to the aforementioned challenges, propose InstructHOI, which leverages context-aware instructions to direct LFM in performing multi-modaa reasoning for HOI detection, as depicted in Fig. (bb). Specifically, for pretrained LFM, first perform HOI-domain fine-tuning bridge the inherent knowledge gap between general and HOI domains
(55 and enhance its interaction reasoning capability, using a light-weight strategy, i.e., LoRA (6). Due the limited availability of HOI reasoning data [25]. we created a large-scale dataset containing 140K image-text pairs by aggregating five existing image-only HOI datasets and transforming the one-hot labels into interaction-reasoninng conversations. Then, we develop Context-aware Instruction Generator (CIG) to guide interaction reasoning. Unlike traditional language-only instructions (27 228) CIG first mines visual interactive context (i.e., appearance and spatial context) at the human-object level. Next, the visual context projected into linguistic space using a two-layer instruction projector, and then is fused with linguistic instructions, providing pair-level context guidance for multi modal interaction reasoning. Furthermore, an Interest Token Selector (ITS) adopted adaptively filter informative image tokens based on the context-aware instructions and reorganize the reasoning token sequences, thereby aligning the reasoning process with interaction regions.
In this paper, our motivation is to explore the potential reasoning capability of LFMs to improve HOI detection. Unlike previous LFM-based approaches, our work directly leverages tailored instruc- tions to guide LFM in facilitating multi-modal reasoning, thereby achieving open-world interaction recognition. Besides, we enhance traditional linguistic instructions by incorporating visual interactive context at the homan-object level, thus providing leir-level multi-modal reasoning guidance. To summarize, our contributions are as follows:
For a pretrained LFM, to bridge the gap between general and HOI-domain knowledge, we build a high-quality interaction-reassninng dataset and perform supervised fine-tuning using lightweight strategy.

---

We develop Context-aware Instruction Generator (CIG) to enhance linguistic instructions by incorporating informative visual context at the human-object level, providing reasoning guidance.
To align the reasoning process with interaction regions, an Interest Token Selector (ITS) is adopted to adaptively filter and reorganize reasoning token sequences based on context-aware instructions.
We evaluate our InstructHOl on two benchmarks: HICO-DET and V-COCO, and it outper- forms the state-of-the-e methods, achieving superior performance both supervised and zero-shot settings.
Related Work
Traditional HOI detectors can be primarily classified into one-stage and two-stage approaches. One-stage methods regard HOI detection as multi-task learning, aiming to simultancously perform object detection and interaction prediction. Earlier methods [29 3
Traditional HOI Detectors:
typically adopt a multi-branch CNN architecture for parallel human-object instance localization and interaction recognition. Then, some auxiliary priors (e.g., interaction points 4 and union boxes 300 are introduced to align instances with their corresponding interactions. Recently, Transformer-based methods 7 6 31 take prominent position, due their exceptional context capture ability. However, such disentangled architecture may suffer from insufficient context exchange between the branches, leading to inferior prediction performance.
Two-stage methods treat HOI detection as two sequential sub-tasks. They initially localize human- object instances with an off-the-shelf detector and then identify interactions leveraging the visual fea- tures extracted from the instance regions. The early extract rich visual interaction representations, e.g., spatial relationship [37]. gaze attention
methods [32
33 34
35
36
9
strive
and
pose feature [33 to assist HOI detection. Recent Transformer -baseed methods 38
39
40
attempt to improve the vanilla Transformer for enhancing feature extraction of HOI detection. De- spite significant efforts in feature extraction strategies and architecture enhancements, accurately distinguishing complex HOIs in open world remains challenging when relying solely on visual representation learning.
LFM-based HOI Detectors: LFM-based HOI detectors can be primarily classified into VLM-style and LLM-style. To further explore discriminative HOI representations, recent approaches [4] 44 45
seek extract prior knowledge from VLMs 46) 47 by leveraging their distinctive ability unify visual and linguistic features. Among VLM-style methods, the pioneering works 44 19 transform one-hot labels into annotation texts via static prompt template, e.g.. "a photo of person [action] a/an [object]". These annotations are then encoded as linguistic priors using CLIP, enabling category-level knowledge transfer. addition, MP-HOI 4 utilizes extra visual prompts to provide fine-grained visual priors, and aims to eliminate the ambiguity in linguistic descriptions. Furthermore, CMMP 21 introduces learnable multi-modal prompts, facilitating the adaptive transfer of semantic knowledge from CLIP at the instance level.
typically
LLM-style methods 22 23 24 48 usually employ language foundation models to generate finer- grained descriptive texts as interactive prompts, which can transfer the generalizable knowledge from LLMs to improve HOI detection. E.g., UniHOI [2 designs a knowledge retrieval process for ChatGPT to acquire comprehensive explanations for each HOI category, which provides rich contextual information for interaction prediction. CMD-SE (4 introduces a two-step GPT-querying mechanism to produce descriptions of human body, and thus generate general body-part prompts, which helpful for recognizing ambiguous actions. However, the existing LFM-based methods primarily focus on transferring HOI-specific knowledge, but fail to explore the reasoning capabilities of LFMs, leading to incomplete exploration of their full potential, particularly for open-world interaction recognition.
3

---

Muiti-Modal Reasoning
Context-Aware Instruction Generator
Interest Token Selector
8,0.8
Map
Instruction
Match (+)
Detector
(+)
T
Projector
LFM
spatial appearance
SeltAttn (+)
Irput Image
linguistic instruction:
FIM
T
MLP ())
Racaon the interactiona betwean humans and ofjects.
Tokenizer Embedding
Tt
LORA
Mir
&
hp
HOI prediction
LFM
Pain-2:
Dymamic Spoit
mage
S2
Pain-t:
Similarty
Human skiing skilowed
Docccio
I(S Sz)
Figure 2: InstructHOI involves two interaction prediction branches: the multi-modal reasoning branch and the visual interaction decoder. The former includes Context-awer Instruction Generator (CIG), Interest Token Selector (ITS), and Multi-Modal Reasoning (MMR). CIG combines visual context with linguistic instructions to generate contexl-aware instructions T.n (Sec. (.33. ITS then adaptively filters image tokens and reorganizes the reasoning token sequences Ty to align the reasoning process with interaction regions (Sec. 3.4). Finally, the LFM's language model in MMR, fine-tuned with LoRA, uses Ty to conduct multi-modal reasoning. generating pair-level interaction texts (Sec, (.5). Mcanwhile, the visual interaction decoder utilizes pair and global image features to perform interaction decoding (Sec. 3.5).
3
Method
3.1
Overall Architecture
The overall architecture of InstructHOI is illustrated in Fig. Given image I, off-the-shelf object detector (i.e., DETR 44)) first employed to localize human and object instances (B₁, B.), and then obtain the Human- Human-Object (H-O) pair features Vn&oo by concatenating the instance features from DETR for each H-O pair. Mcanwhile, following the dynamic image encoding strategy in [50], we dynamically split the image and obtain the global and local images (Lglobals Liocal), which then separately encoded into the global and local image features (Vg. Vi), using the pretrained image encoder of LFM (i.e., Intern VL.2 (50).
InstructHOI involves two interaction prediction branches: the multi-modal reasoning branch and the visual interaction decoder. The former branch mainly includes three components: Context-aware Instruction Generator (CIG), Interest Token Selector (ITS), and Multi-Modal Reasoning (MMR). To direct LFM in facilitating multi-modal reasoning and achieving pair-level interaction prediction, CIG first extracts the appearance and spatial context embedding T, of each H-O pair, which is then inserted into linguistic instructions. forming pair-specific context-aware instructions T., (Sec. 3.3) Furthermore, to align reasoning process with interaction regions. ITS filters informative tokens from local image tokens based on the instructions T... and then reorganizes them into reasoning token sequences T, (Sec. 3.4). The LFM's s language model in MMR, fine-tuned with LoRA, utilizes the filtered token sequences Ty to achieve pair-level interaction reasoning and acquire interaction- reasoning probability distribution S, (Sec. 3.5). Meanwhile, in the visual interaction decoder, the pair features Visco act as Query, while the global image features Vg act as Key and Value, performing interaction decoding and yielding interaction-decodingg probability distribution Si (Sec. 3.5) Finally, both distributions S and S2 are combined to yield the final interaction score.
HOI-domain Fine-tuning
Different from task-specific models, Large Foundation Models (LFMs) are typically pretrained on vast and diverse datasets, acquiring general-domain knowledge across both visual and linguistic modalities. However, such general models often struggle to achieve accurate zero-shot interaction prediction, due to the gap between general knowledge and that specific to HOI domain (25). To bridge the knowledge gap and enhance the interaction reasoning capability for HOI detection,

---

Pair 1: visual linginsti
Insiruciins
The
reprasantative token
Match
SelfAttn
Global Images
T in
T
M
Tg
Token Selection
T!
MLP (
Local Images
T
Sequence Reorganization
Matrix Construction
Figure 3: The illustration of Interest Token Selector. It contains three steps: token selection, matrix construction and sequence rcorganization.
introduce a multi-modal LFM (i.e., Intern V1.2) and conduct HOI-domain fine-tuning on generated high-quality dataset.
Specifically, we employ a light-weight adaptation strategy (i.e., LoRA) facilitate efficient and low-consumption fine-tuning. As shown in Fig. - we freeze the entire pretrained LFM model, including the image encoder, the Vision-Language MLP (V-L MLP), and the language model, while only training few injectable parameters (about 0.8% of the pretrained model's parameters) for the language model of InternVL2, acquiring fine-tuned model T Additionally, due to the limited availability of HOI reasoning data [25]. we aggregate five existing image-only HOI 55 and build a high-quality dataset containing 140K image-text pairs across KK object categories, 600 action categories, and 15K HOI categories. To acquire interaction conversations, E.g.. for
datasets [5
52 5
reasoning texts, we transform original one-hot labels into Question& buman-bike pair with 'sit" and 'ride' interactions, the conversation is formulated as: (Question: Reason the interaction relationships between humans and objects in the image; Answer: Human is sitting and riding bike). Moreover, to avoid potential data leakage during evaluation, all testing data used in the experiments are excluded from the one-tuning dataset. The visualization of the dataset is presented in the Supplementary Material.
3.3 Context-aware Instruction Generator (CIG)
explore the potential reasoning capabilities of LFMs for tackling complex visual tasks using different strategies, e.g., chain-of-thougll language instruction [27). However, reasoning about the ambiguous interactions in complex scenarios needs accurate guidance assist LFM in understanding the spatial relationship and instance appearance of each H-O pair, while language-only instructions can hardly provide these visual context clues. To remedy this, we develop Context-aware Instruction Generator (CIG), which incorporates the informative visual context of each H-O pair in linguistic instructions, providing multi-mode guidance for pair-level interaction reasoning.
Recent studies [56 57
As shown in Fig. we first derive the pair features Vnko by combining the instance features for each H-O pair, which are extracted from the hierarchical backbone of the object detector and contain discriminative characteristics of each instance. Then, the multi-level Vnãoo are flattened and concatenated, serving as the visual appearance representations of each H-O pair. To further encourage LFM to be aware of spatial relationships, we encode H-O spatial context based on the localization boxes B₁ & B,, Following the previous works (58 21). we extract spatial features from B, & B., by calculating the intersection-over-uniee appearance and spatial features are mapped to dimensionality of d, forming the visual context embedding T.. The process can be formulated as:
scaled distance, spatial direction, etc. Subsequently, both
T. Map (SPEnc (Bn, Bo), Vi,,o)
(I)
where SPEne(-) represents spatial feature encoding, and Map(-) indicates feature mapping operation. Meanwhile, the linguistic instructions are also encoded into linguistic context embedding T, using the tokenizer of VL2.
Inspired by the "ViT > V-L MLP - LLM" architecture in the existing studies (e.g., LLaVA (59 we leverage two-layer instruction projector to map the visual context embedding T, into the linguistic embedding space, eliminating the knowledge gap between visual and linguistic modalities. Finally, the pair-level visual context embedding T, is fused with the linguistic context embedding T., forming
5

---

pair-level context-and -ware instructions Tin. Overall, the process can be formulated as:
Tin = Concat (Proj, (Te), T,),
(2)
where (-) indicates the instruction projector, which consists of a two-layer MLP followed by GeLU layer and Concat(-) denotes the concatenation operation.
3.4 Interest Token Selector (ITS)
According to the dynamic image encoding strategy [50]. input image is typically dynamically split, acquiring global and local images (Igtobals Iocall) ) where the global image gloobel provides holistic image context, and the local images Local offer region-level context information. Considering that, within an image, the interactive regions of H-O pairs also dynamically vary based on their locations and interaction categories. E.g. for (human, hold, apple), the interactive region mainly focuses on the hand regions, while for (human, kick, football,,, the interactive region shifts to the leg regions. To align pair-level reasoning with the corresponding interaction regions, we develop an Interest Token Selector (ITS), which evaluates the interaction relevance of local images Locol for each H-O pair, based on the context-aware instructions Tr. Thus, ITS can adaptively select the informative image tokens and then reorganizes the reasoning token sequence for each H-O pair.
Following the multi- modal reasoning mechanism in Intern VL2, the image features (Vi, Vg) are projected into linguistic space using the V-L MLP of Intern VL2, acquiring image tokens (Ti, Tg). Additionally, as depicted Fig. B the Interest Token Selector contains three steps: token selection, matrix construction and sequence reorganization. Firslly, to extract the representative tokens from the instructions Tin, we calculate the cosine similarity between Tin and global image tokens Tg. and select the top-n most similar ones as representative instruction tokens in. as follows:
T Match (g) E RR. Xxxxd
where Match(-) represents the cosine similarity operation and selection, N₂ and indicate the number of H-O pairs and token dimension, respectively. Afterwards, self-attention layer is employed to facilitate feature fusion and context propagation between Tin and the local image tokens Ti. An MLP then applied to predict the interaction relevance of local images for each H-O pair, constructing interaction-relevnnce matrix Mir, as follows:
Mir MLP(Sel(Aun(Q, K,V V concat Ti))) E IR.,
where SelfAttn(.) represents self-attention layer and N₁ indicates the number of local images.
Finally, we use softmax operation to calculate the relevance probability distribution based on the matrix Mir, subsequently selecting the informative tokens from 7i and reorganizing the reasoning token sequence T for each H-O pair in the format of [[selected local images tokens), {global image tokens), (instruction tokens)] as follows:
Ty Concat(Ti, T,, Tin), where 7, = Filter (Ti, Mar),
(5)
Ti represents the selected local images tokens and Filter(-) indicates the image token filtering operation based on the interaction-relevance matrix. The visualization of ITS are provided in the Supplementary Material.
3.5 Inference and Training
The inference process is illustrated in Fig. 2 InstructHOI involves two interaction prediction branches: the visual interaction decoder and the multi-modal reasoning branch. In the visual interaction decoder, the interaction decoding is performed based on the visual representations, taking pair features Vnto as Query and global image features V, as Key and Value, and yields interaction-decodingg probability distribution Si:
Inference.
S (Crosssstt((( Vnko: K,V Vg)) E R.N.N.
(6)
where CrossAAuu()) indicates the cross-attention operation, and Proja()) represents the distribution projector, which consists of MI.P followed by sigmoid operation, and N, represents the number of interaction categories.

---

For the multi-modal reasoning branch, the fine-tuned InternVL2, n utilizes the refined reasoning token sequence y to conduct multi-modal reasoning and generate pair-level interaction descrip- tions. Next, we calculate textual cosine similarity between Lpp and textual HOI labels Lu, and then compute miteroction-rasooning probability distribution S2, following 43
2 = Softmax (Feos L.), where Lp = T tora (Ty)
(7)
where (-) indicates the cosine similarity operation, and S2 E IRN,XN, Finally, the total interaction prediction score obtained by combining distributions Si and S2:
Shoi = (Sx)** ^ (S.)* S1 S2.
(8)
where Sn, and S, indicate the detection scores human and object instances from the object detector, respectively.
Training. To supervise the visual interaction decoder, we employ the following Focal Loss:
No
N
c. =c Vic
FocalLoss(y ,1.
where yhe E } in y indicates whether the groundtruth of the i-th human-object pair contains the c-th interaction class and S in S1 is the corresponding predicted probability. In addition, to supervise the multi-modal reasoning branch, we adopt similarity constraint loss, as follows:
1
Σ yhe c(i,c) Σn ((i.j)
Srim
log
where Z(i,j) exp(F cas coo (L, L()).
(10)
Np
Overall, the total loss function is formulated as: L = Lv + aLsim.
4 Experiments
4.1 Experimental Setting
Datasets.
Following previous works, we conduct experiments on two commonly used HOI datasets:
HICO-DET (51] and V-COCO [52]. The HICO-DET dataset comprises 47776 images, with 38115 for training and 9658 for testing, covering 117 actions, 80 objects, and 600 HOIs. Additionally, the 600 HOIs are divided into 138 Rare and 462 Non-Rare categories based on the sample distribution. The V-COCO dataset, contains 10346 images, including 5400 in the trainval set, and 4946 in the test. set, across 29 actions, 80 objects, and 259 HOIs.
Evaluation Metric. Following the standard metric, the mean Average Precision (mAP) is adopted to evaluate the performance of InstructHOL. During the evaluation, a true positive HOI triplet needs to meet two criteria: 1) the predicted human and object bounding boxes should have Intersection over Union (IoU) values greater than 0.5 with the ground truth, and 2) the HOI classification is correct.
Implementation Details. We take the DETR for object detection and adopt the pretrained Intern VL.2 as the foundation model. During training, we freeze the external models (DETR and Intern and update the parameters of the remaining components in InstructHOL. The entire InstructHOI model is trained on four Tesla A800 GPUs with a batch size of 16 for epochs, using the AdamW [60] optimizer.
4.2 Comparisons with the State-of-the-Arts
4.2.1 Supervised Setting
In Table we present the quantitative results for the supervised setting on the HICO-DET and V-COCO datasets, respectively. Notably, our method outperforms all the existing state-of-the-a methods on both datasets. For the HICO-DET dataset, InstructHOI achieves remarkable mAPs of 47.68 and 49.89 in the default and known object full settings. Compared to recent state-of-tth-arr HOI detectors RLIPv2 [61] and Pose-Aware (62), our model obtains significant performance gains of 2.59 mAP (relatively 5.74%) over RLIPv2 and 1.67 mAP (relatively 3.63%) over Posc-Aware,
7

---

Table I: Performance comparison HICO-DET and V-COCO datasets, For results on HICO-DET, we follow commonly used experimental setting to fine-tune the object detector its training set.
HCCO-DET
V-COCO
Default
Known Object
Method
Backbone
Full
Rare
Non-Rare
Full
Rare
Non-Rare
AP
AP#2
One-stage methods
FGAHOI [33
Swin-L. Swin-I.
37.18
30.71
39.11
38.93
31.93
41.02
RI.IPv2
45.09
43.23
45.64
72.1
Two-stage methods
PVIC 64
Swin-L
44.32
44.61
44.24
47.81
48.38
47.64
64.1
70.2
Pose-Aware
62
Swin-I.
46.01
46.74
45,80
49.50
50.59
49.18
63.0
68.7
LFM-based methods
EZ-H101 48
R50+
38.61
37.70
38.89
60.5
66.2
UniHOI-1 22
RITI+V
40.95
40.27
41.32
43.26
43.12
43.25
68.1
70.8
HII 65
VIT-L
42.54
42.95
42.35
44.91
45.18
44.83
67.1
71.1
MP-HO1 43
Swin-L+ViT
44.53
44.48
44.55
66.2
67.6
SICHOI 23
RI0I+VIT-L.
45.04
45.61
44.88
48.16
48.37
48.09
71.1
75.6
InstrucHHO (Ours)
R50+ VIT-
45.95
46.51
45.78
48.57
49.23
70.8
74.2
InstrucUIOO (Ours)
VIT-+
47.68
47.97
47.59
49.89
50.92
49.58
72.4
76.1
respectively. By introducing HOI-domain LFM, InstructHOI can extract rich contextual clues and discriminative interaction representations, tackle ambiguous interaction detection in complex scenarios. Additionally, compared to the state-of-the-art LFM-based methods SICHOI 2 and MP-HOI 431, InstructHOI also achieves significant performance improvements, outperforming SICHOI by 2.64 mAP (relatively 5.86%) and MP-HOI by 3.15 mAP (relatively 7.07%), respectively, in the commonly used default full setting. Specifically, VLM-style methods (e.g., ADA-CM and MP-HOI) adopt single- or multi-moda prompts to transfer HOI-specific knowledge from LFM, while L.LM-style approaches (e-g-. UniHOI and SICHOI) generate comprehensive descriptions as interactive prompt based on language foundation models. However, all these LFM-based methods primarily focus on transferring HOI-specific knowledge, without exploring the potential reasoning capabilities of LFMs. Unlike existing knowledge transfer methods, InstructHOI directly leverages context-aware instructions to guide LFM facilitating pair-level multi-modal reasoning, acquiring discriminative interaction representations for ambiguous and open- world interaction recognition.
For V-COCO dataset, as reported in the right part of Table I InstructHOI also performs the best among all the state-of-the-ard methods, achieving A role of 72.4 in scenario #1 and A of 76.1 in role scenario #2. Specifically, comparing the recent HOI detectors RLIPv2 and SICHOI, InstructHOI demonstrates superior performance. e.g.. AP#1 rale of 72.4 vs 72.1 and 71.1, and AP#2
rafe of 76.1 vs 74.1 and 75.6. Even using "Resnet50" backbone, our method still performs better than most of the state-of-the-art approaches. The superiority of our proposed InstructHOI comes from the fact that we fully exploit the reasoning ability of large foundation models rather than simply transferring knowledge.
4.2.2 Zero-shot Setting
Consistent with previous zero-shot experiments 17
66
22 we evaluate our method on HICO-DET
under four zero-shot settings: 1) Rare First Unseen Combination (RF-UC) constructs training set with all the object and verb categories but excludes a certain number of rare HOI categories. 2) Non-rare First Unseen Combination (NF-UC) prioritizes non-rare interactions when selecting the held-out HOI categories. 3) Unseen Object (UO) is designed assess interaction recognition with novel object categories. 4) Unseen Verb (UV) focuses on discovering novel action categories. For fair comparison, we present recent LFM-base zero-shot HOI detectors with the same "ResNet50" backbone in Table where our InstructHOI surpasses all other methods across four zero-shot settings.
For RF-UC and NF-UC settings, InstructHOI achieves 36.82 mAP and 36.42 mAP for unseen HOI categories, respectively. Compared to the latest method SICHOI, our approach achieves gains of 3.14 mAP and 2.58 mAP in the RF-UC full and unseen settings, respectively. as well as gains of 2.59 mAP and 1.90 mAP in the NF-UC full and unseen settings, respectively. The reason is that our proposed InstructHOO has interactive reasoning capabilities, rather than simply borrowing general knowledge from the LFMs. As for UO and UV settings, InstructHOl attains mAPs of 39.92 and 31.64 for unseen HOI categories, respectively. Compared to the latest method CMMP, our approach achieves improvements of 0.79 mAP and 0.25 mAP in the UO full and unseen settings, respectively,

---

Table 2: Zero-shot generalization HICO-DET 51
Table 3: Performance contribution of each component.
Mothods
Type
Full
Soon
Unscen
HICO DET (Default)
V.COCo
DiffusionHOI 665
RE-UC
35.89
36.77
32.06
Method
Full
Rare
Non-Rare
AP
AP
EZ-HOI 48
RF-UC
36.73
37.35
34.24
Base
36.21
32.84
37.22
64.8
70.1
CMMP 21
RE-UC
37.13
37.42
35,98
+MMR
43.06
43.40
42.96
68.9
72.4
SICHOI
RF-UC
40.11
41.58
34.24
+MMR+CIG
44,98
44.69
45.07
70,2
73.9
+MMR+CIGHHTS
45.95
46.51
45.78
70.8
74.2
InstructHOI
RF-UC
43.25
44.86
36.82
BCOM 67
NF-UC
32.03
31.76
33.12
-ontext-aware Table 4: Effect of Context aware Instruction Generator.
I1OIGen
NF-UC
33.08
32.86
33.98
HICO-DET (Delault)
V-COCO
CMMP 21
NF-UC
35.13
35.53
33.52
Method
SICHOI 23
NF-UC
35.75
36.06
34.52
Full
Rare Non-Rare
AP
Ap
rale
InstructHOI
NF-UC
38,34
38.82
36.42
Basc+MMR
43.06 43.40
42,96
68.9
72.4
UniHOI
22
UO
31,56
34.76
19.72
+SC
43.84 43.87
43.83
69.3
72.9
1101Gen 68
UO
33,48
32.90
36.35
+AC
44.43 44.10
44.53
69.8
73.6
EZ-11O1
36.38
36.02
38.17
+AC+SC
44.98 44.69
45.07
70.2
73.9
CMMP 21
UO
36.74
36.15
39.67
Instruct
UO
37.53
37.05
39.92
Table S: Effect of Interest Token Selector.
HOlGen 68
UV
32.34
34.31
20.27
HICO-DET (Default)
V-COCO
Unil1Ol
2
UV
34,68
36.78
26.05
Image Token
Full
Rare
Non Rare
AP$1 role
AP nole
CMMP
21
UV
36.38
37.28
30.84
43.62
42.29
44.02
69.3
EZ-HOI
48
UV
36.84
38.15
28.82
T,
44.98
44.69
45.07
70.2
73.9
Instrucil
UV
38.12
39.17
31.64
Tg +
46.51
45.78
and 1.74 mAP and 0.80 mAP in the UV full and unseen settings, respectively. All the four zero-shot experimental results consistently demonstrate the effectiveness of our InstructHOI in detecting unseen and novel HOIs. By leveraging the interaction reasoning capabilities of HOI-domain LFMs, InstructHOI exhibits superior open-world interaction detection performance and generalization.
4.3
Ablation Study
In this subsection, we evaluate the effects of Multi-Modal Reasoning (MMR), Context-aware In- struction Generator (CIG), and Interest Token Selector (ITS) components in InstructHOO both the HICO-DET and V-COCO datasets. For a fair comparison, we create a baseline mode (denoted as "Base") by simply combining the DETR (using Resnet50 backbone) and visual interaction decoder branch (using ViT-L as backbone), which represents a degraded version of InstructHOI without MMR, CIG, and ITS. Here, the standalone 'MMR' refers multi-modal reasoning with language-only instructions, while 'MMR+CIG' denotes multi-moue multi- reasoning with context-aware instructions (see subsection B.3). Additional ablation studies on advanced object detector, HOI-domain fine-tuning, and the number of representative tokens are provided in the Supplementary Material.
Component Ablation. As shown in Table 3 each component of InstructHOI significantly enhances the baseline model. Specifically, MMR improves the baseline by 6.85 mAP in full setting on HICO- DET, while CIG provides an additional improvement of 1.92 mAP. Ultimately, the combination of all the three components results in total improvement of 9.74 mAP. The above results demonstrate that the reasoning capabilities of LFM can significantly improve HOI detection, as well as highlight the effectiveness of CIG and ITS in further improving LFM's reasoning abilities.
Context-aware Instruction Generator. Within the Context-aware Instruction Generator (CIG), integrate visual context into linguistic instructions to enhance the spatial and appearance understand- ing of LFM. In Table 4 we evaluate the Spatial Context (SC) and the Appearance Context (AC) in CIG separately, based on the "Base + MMR" model (i.e., reasoning with language-only instructions). The results demonstrate that both AC and SC can enhance the language-only instructions, providing context guidance for interaction reasoning.
Interest Token Selector. As shown in Table 5 evaluate the effectiveness of the Interest Token Selector (ITS) by using different combinations of image tokens: T, (global image tokens), Ti (local image tokens), and T; (selected local image tokens). Specifically, the combination of (T, Ti) outperforms (Tg + TT) by 0.97 mAP the full setting of HICO-DET. This indicates that the ITS effectively filters informative image tokens from Tj, aligning the reasoning with interaction areas.

---

5 Conclusion
In this paper, we propose a novel LFM-based HOI detector, InstructHOL. Different from the existing LFM-based approaches, InstructHOI directly learns tailored instructions to guide LFM in facilitating multi-modal reasoning, and thus can improve the open-world interaction recognition. Specifically, develop Context-aware Instruction Generator (CIG) enhance linguistic instructions by incorporating visual interactive context, forming pair-level reasoning guidance. Furthermore, an Interest Token Selector (ITS) is adopted to align reasoning process with interaction regions. Extensive experiments on two public benchmarks demonstrate that our proposed method outperforms the state-of-the-art ones, under both supervised and zero-shot setrings. Ablation studies also prove the effectiveness of each component in our proposed InstructHOI.
Acknowledgment
This work was supported in part by the National Natural Science Foundation of China under Grants 62206075, 62573163, 62503139, and 62261 60652, in part by the GuangDong Basic and Applied Basic Research Foundation under Grant 2024A151501 2028,, in part by the Shenzhen Science and Technology Program under Grant GXWD20231 1291 25006001,, in part by the Science and Technology Development Fund (FDCT), Macau SAR, under Grant 0095/2022/AFJ.
References
[1] B. Pang, K. Zha, Y. Zhang, and C. Lu, "Further understanding videos through adverbs: A new vidco task," in AAAI Conference on Artificial Intelligence, pp. 1183-11830, 2020.
[2] Y. Liu, W. Chen, Y. Bai. J. Luo, X. Song, K. Jiang. Z. Li, G. Zhao, J. Lin, G. Li, et al., "Aligning cyber space with physical world: A comprehensive survey on embodied ai," arXiv preprint arXiv:2407006886, 2024.
[3] L. Chen, P. Wu, K. Chitta, B. Jaeger, A. Geiger, and H. Li, "End-to-end autonomous driving: Challenges and frontiers," IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 46, no. 12, pp. 10164- 10183, 2024.
[4] Y. Liao, S. Liu, F. Wang. Y. Chen, C. Qian, and J. Feng. "Ppdm: Parallel point detection and matching for real-time human-objec interaction detection," in IEEE Conference on Computer Vision and Pattern Recognition, pp. 482 490, 2020.
[5] T. Wang, Yang, Danelljan, F. S. Khan, X. Zhang, Sun, "Learning human human-object interaction detection using interaction points," in IEEE Conference on Computer Vision and Pattern Recognition, pp. 41 4125,11152222 2020.
[6] M. Tamura, H. Ohashi, and T. Yoshinaga. "Qpic: Query-based pairwise human-object interaction detection with image-wide contextual information," IEEE Conference on Computer Vision and Puttern Recognition. pp. 10410-10419, 2021.
[7] C. Zou, B. Wang, Y. Hu, J. Liu, Q. Wu, Y. Zhao, B. Li, C. Zhang, C. Zhang, Y. Wei, et al., "End-to-end human object interaction detection with hoi transformer," in IEEE Conference on Computer Vision and Pattern Recognition, pp. 11825 11834, 2021.
[8] T. Gupta, A. Schwing, and D. Hoiem, "No-frills human-object interaction detection: Factorization, layout encodings, and training techniques." in IEFE International Conference Computer Vision, pp. 9677-9685, 2019.
[9] C. Gao, J. Xu, Y. Zou, and J.-B. Huang, "Drg: Dual relation graph for human-object interaction detection," in European Conference on Computer Vision, pp. 696-712, Springer, 2020.
[10] J. Park, J. Park, and J.SS. Lec, "Viplo: Vision transformer based pondicondi self-loop graph for human-object -ojecct interaction detection," " IEEE Conference on Computer Vision and Pattern Recognition, pp. 17152-17162, 2023.
[11] B. Xu. J. Li. Y. Wong, Q Zhao, and M. S. Kankanhalli, "Interact as you intend: Intention -rivvn human- object interaction detection," IEEE Transactions on Multimedia, vol. 22, no. 6, pp. 4423-143 2019.
10

---

[12] L. Zhang, W. Suo, P. Wang, and Y. Zhang. "A plug-and-play method for rare human-object interactions detection by bridging domain gap," in ACM International Conference Multimedia, pp. 8613-8622, 2024.
[13] Y. Zhang, Y. Pan, Yao, R. Huang. T. Mei, and C.-W. Chen, "Exploring structure-aware transformer over interaction proposals for human-object interaction detection," in IEEE Conference on Computer Vision and Pattern Recognition, pp. 19548 19557,, 2022.
[14] J. Lim, V. M. Baskaran, J. M.-Y. Lim, K. Wong, J. Sec, and M. Tistarelli, "Emett: An efficient and reliable human-object interaction detection network," IEEE Transactions on Image Processing, vol. 32, pp. 964-979, 2023.
[15] H. Yuan, J. Jiang, S. Albanic, T. Feng, Z. Huang. D. Ni, and M. Tang, 'Rlip: Relational language-imag pre-training for human-object interaction detection," Advances in Neural Information Processing Systems, vol. 35, pp. 31.31-3733333 2022.
[16] Z. Li, X. Li, C. Ding, and X. Xu, "Disentangled pre-training for human-object interaction detection," in IEEE Conference on Computer Vision and Pattern Recognition, pp. 28191-28201, 2024.
[17] Y. Liao, A. Zhang, M. Lu, Y. Wang, X. Li, and Liu, "Gen-vlkt: Simplify association and enhance inter- action understanding for hoi detection," in JEEE Conference on Computer Vision and Pattern Recognition, pp. 20123-20132, 2022.
[18] A. Iftekhar, H. Chen, K. Kundu, X. Li, J. Tighe, and D. Modolo. "What to look at and where: Semantic and spatial refined transformer for detecting -ojaet interactions." in IEEE Conference on Computer Vision and Pottern Recognition. pp. 5353-5363, 2022.
[19] X. Qu, C. Ding, X. Li, X. Zhong, and D. Tao. "Distillation using oracle queries for transformer bassed human human-object interaction detection," in IEEE Conference on Computer Vision and Puttern Recognition, pp. 19558-19567, 2022.
[20] A. Radford. J. W. Kim, C. Hallacy. A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, al., "Learning transferable visual models from natural language supervision." in International Conference on Machine Learning, pp. 8748-8763 PMLR, 2021.
[21] T. Lei, S. Yin, Peng, and Y. Liu, "Exploring conditional multi-modal prompts for zero-shot hoi detection," in European Conference on Computer Vision. pp. 1-19, Springer, 2025.
[22] Y. Cao, Q. Tang, X. Su, S. Chen, S. You, X. Lu, and C. Xu, "Detecting any human-object interaction relationship: Universal hoi detector with spatial prompt learning on foundation models," Advances in Neural Information Processing Systems, vol. 36, pp. 739-751, 2023.
[23] J. Luo, W. Ren, W. Jiang, X. Chen, Q. Wang, Z. Han, and H. Liu, "Discovering syntactic interaction clues for human-ohject interaction detection," IEEE Conference on Computer Vision and Pottern Recognition, pp. 28212-28222, 2024.
[24] T. Lei, S. Yin, and Y. Liu, "Exploring the potential of large foundation models for opcn-wocabulary hoi detection," in IEEE Conference Computer Vision and Pattern Recognition, pp. 16657-16667, 2024.
[25] J. Gao, C. Cai, R. Wang. W. Liu, K.-H. Yap. K. Garg, and B.-S. Han, "Cl-hoi: Cross-level hbjeet interaction distillation from vision large language models," arXiv preprint arXiv:2410.15657 2024.
[26] E. J. Hu, yelong shen, P. Wallis, Allen-Zhu, Li. Wang, L. Wang, and W. Chen, "LoRA: Low-rank adaptation of large language models." in International Conference on Learning Representations, 2022.
[27] Y. Hu, O. Stretcu, C.-T. Lu, K. Viswanathan, K. Hata, E. Luo, R. Krishna, and A. Fuxman, "Visual program distillation: Distilling tools and programmatic reasoning into vision-language models," in IELE Conference Computer Vision and Pattern Recognition, pp. 9590-9601, 2024.
[28] S. Lee, W. J. Kim, J. Chang, and C. Ye, "LIM-CXR: instructionfinettnee L.M for CXR image understanding and generation." in International Conference on Learning Representations, 2024.
[29] X. Wu, Y.-L. Li, X. Liu, J. Zhang, Y. Wu, and C. Lu, "Mining cross- cross-person cues for body-part interactiveness learning in hoi detection," " in European Conference on Computer Vision, pp. 121-136, Springer, 2022.
[30] B. Kim, T. Choi, J. Kang, and H. J. Kim, "Uniondet: Union-level detector towards cale human- object interaction detection," " in European Conference Computer Vision, pp. 498 088-514 514, Springer, 2020.
11

---

[31] D. Zhou, Z. Liu, Wang, L. Wang, T. Hu, E. Ding, and J. Wang, "Human-object interaction detection via disentangled transformer," in IEEE Conference Computer Vision and Pattern Recognition, pp. 19568- 19577, 2022.
[32] A. Bansal, S. S. Rambhatla, Shrivastava, and R. Chellappa. "Detecting human-object interactions via functional generalization, in AAAI Conference on Artificial Intelligence, vol. 34, pp. 10460-10469, 2020.
[33] D.J. Kim, X. Sun, J. Choi, S. Lin, and I. S. Kweon, "Detecting human-object interactions with action co-occurrence priors," in European Conference on Computer Vision, pp. 718-736, Springer, 2020.
[34] X. Zhong, C. Ding. X. Qu, and D. Tao, "Polysemy deciphering network for robust human object interaction detection," International Journal of Computer Vision, vol. 129, pp. 1910-1929, 2021.
[35] O. Ulutan, A. S. M. Iftckhar, and B. S. Manjunath, "Vsgnet: Spatial attention network for detecting human object interactions using graph convolutions," in IEEE Conference on Computer Vision and Puttern Recognition, June 2020.
[36] H. Wang. W.-s. Zheng, and L. Yingbiao, "Contextual heterogeneous graph network for human-object interaction detection," in European Conference on Computer Vision, pp. 248 248-264, Springer, 2020.
[37] Y. Liu, Q. Chen, and A. Zisserman, "Amplifying key cues for couman-object-oe interaction detection," in European Conference on Computer Vision, pp. 248-265, Springer, 2020.
[38] B. Kim, Mun, K.-W. On, M. Shin, J. Lee, and F.-S. Kim, "Mstr: Multi-scale transformer for end-to-end human-object interaction detection," in JEEE Conference on Computer Vision and Puttern Recognition, pp. 19578-195555 19987, 2022.
[39] X. Liu, Y.-L. Li, X. Wu, Y.-W. Tai, C. Lu, and C.-K. Tang, "Interactiveness field in human-object interactions," in JEEE Conference Computer Vision and Pattern Recognition. pp. 201 2012020222220 2022.
[40] C. Xie, F. Zeng, Y. Hu, S. Liang, and Y. Wei, "Category query learning for human-object interaction classification." in IEEE Conference Computer Vision and Pattern Recognition, pp. 15275-15284, 2023.
[41] F. Z. Zhang, D. Campbell, and S. Gould, "Efficient two stage detection of human- object interactions with novel unary-parne pairwise transformer," in IEEE Conference on Computer Vision and Puttern Recognition, pp. 20104-20112, 2022.
[42] L. Li, J. Wei, W. Wang, and Y. Yang, "Neural-logic human-object interaction detection," " Advances in Neural Information Processing Systems, vol. 36, 2024.
[43] J. Yang, B. Li, A. Zeng. L. Zhang, and R. Zhang. "Open-world numan-object interaction detection via multi- modal prompts," in IEEE Conference on Computer Vision and Pattern Recognition, pp. (6954-16966 2024.
[44] S. Ning, L. Qiu, Y. Liu, and X. He, "Hoiclip: Efficient knowledge transfer for hoi detection with vision- language models," in IEEE Conference on Computer Vision and Pattern Recognition, pp. 23507-23517, 2023.
[45] S. Zheng, B. Xu, and Q. Jin, "Open-category human-object interaction pre-training via language modeling framework." in IEEE Conference on Computer Vision and Pattern Recognition, pp. 19392-19403 19402, 2023.
[46] J. Li, D. Li, C. Xiong, and S. Hoi, "Blip: Bootstrapping language-imas pre-training for unified vision- language understanding and generation." in International Conference on Machine Learning, pp. 12888- 12900, PMLR, 2022.
[47] J. Li, D. Li, S. Savarese, and S. Hoi, 2Blip-2: Bootstrapping banguage-image pre-training with frozen image encoders and large language models," " in International Conference on Machine Learning, pp. 19730-19742, PMLR, 2023.
[48] Q Lei, B. Wang, and T. Robby T., "Ez-hoi: Vlm adaptation via guided prompt learning for zero-shot hoi detection," Advances in Neural Information Processing Systems, 2024.
[49] N. Carion. Massa, G. Synnaeve, N. Usunier, A. Kirillov, and S. Zagoruyko. "End-to-end object detection with transformers," " in European Conference on Computer Vision pp. 213-229, Springer, 2020.
[50] Z. Chen, W. Wang, H. Tian, S. Ye, Z. Gao, E. Cui, W. Tong. K. Hu, J. Luo, Z. Ma. et al., "How far are we to gpt-4y? closing the gap to commercial multimodal models with open-soure suites," arXiv preprint 204822404.16621,, 2024.
12

---

[51] Y.-W. Chao. Y. Liu. X. Liu, H. Zeng, and J. Deng. "Learning to detect human-object interactions," in IEEE Winter Conference Applications of Computer Vision. pp. 381-389, 2018.
[52] S. Gupta and J. Malik, "Visual semantic role labeling," arXiv preprint arXiv: 1505.04474, 2015.
[53] S. Wang, K.-H. Yap. H. Ding, J. Wu, J. Yuan, and Y.-P. Tan, "Discovering human interactions with large- vocabulary objects via query and multi-scale detection," in IEEE International Conference Computer Vision, pp. 13475-13484, 2021.
[54] A. Kuznetsova, H. Rom, N. Alldrin, J. Uijlings, I. Krasin, J. Pont-Tuset, S. Kamali, S. Popov, M. Malloci, A. Kolesnikov, al., "The open images dataset v4: Unified image classification, object detection, and visual relationship detection at scale," International journaa of computer vision, vol. 128, no. 7, pp. 1956-1981, 2020.
[55] Y.-.. Li, Xu, Huang. X. Liu, Z. Ma. Chen, Wang. H.-S. Fang, and C. Hake, "Human activity knowledge engine," arXiv preprint arXiv:1904.0000 06539, vol. 2. no. 6, 2019.
[56] X. Wu, Y.-L. Li, J. Sun, and C. Lu, "Symbol-IIm: leverage language models for symbolic system visual human activity reasoning," Advances in Neural Information Processing Systems, vol. 36, 2024.
[57] Z. Liu, Hu, Zhang, H. Guo, S. Ke, B. Liu, and Wang. "Reason for future, act for now: A principled architecture for autonomous Ilm agents." in International Conference on Machine Learning, 2023.
[58] Jiang. W. Ren, Tian, L. Qu, Wang, and H. Liu, 'Exploring self-and cross-triplet correlations for human-object interaction detectionn"" in AAAI Conference on Artificial Intelligence, vol. 38, pp. 2543-2551, 2024.
[59] H. Liu, C. Li, Q. Wu, and Y. Lee, "Visual instruction tuning," " Advances in neural information processing systems, vol. 36, pp. 34892-34916, 2023.
[60] Loshchilov and E. Hutter, "Decoupled weight decay regularization," arXiv preprint arXiv: 171.05111, 2017.
[61] H. Yuan, S. Zhang, X. Wang. S. Albanie, Y. Pan, T. Feng, J. Jiang, D Ni, Y. Zhang, and D. Zhao, "Rlipv2: Fast scaling of relational language-image pre-training," in IEEE International Conference Computer Vision, pp. 21649 21661. 2023.
[62] E. Z. Wu, Y. Li, Y. Wang, and S. Wang. "Exploring pose-aware human-object interaction via hybrid learning." in IEEE Conference on Computer Vision and Pattern Recognition, pp. 17815-17825, 2024.
[63] S. Ma, Y. Wang. S. Wang, and Y. Wei, "Fgahoi: Fine-grained anchors for human-object interaction detection," IEEE Transactions Pattern Analysis and Machine Intelligence, vol. 46, no. 4, pp. 2415-2429, 2024.
[64] F. Z. Zhang, Y. Yuan, Campbell, Z. Zhong, and Gould, "Exploring predicate visual context in detecting of human-object interactions," in IEEE International Conference on Computer Vision, pp. 10411 1-10421, 2023.
[65] L. Li, W. Wang, and Y. Yang, -ouean-object interaction detection collaborated with large relation-driven diffusion models," Advances in Neural Information Processing Syxtems, 2024.
[66] T. Lei, F. Caba, Q. Chen, H. Jin, Y. Peng, and Y. Liu, "Efficient adaptive human-object interaction detection with concept-guided memory," in IEEE International Conference on Computer Vision, pp. 6480-6444 -480- 2023.
[67] G. Wang. Y. Guo, Z. Xu, and M. Kankanhalli, "Bilateral adaptation for human-object interaction detection with occlusion robustness." in IEEE Conference on Computer Vision and Pattern Recognition, pp. 27970- 27980, 2024.
[68] Y. Guo, Y. Liu, J. Li, W. Wang, and Q. Jia. "Unseen more: Unlocking the potential of clip for generative zero-shot hoi detection," in ACM International Conference on Multimedia, pp. 1711-1720, 2024.
13

---

NeurIPS Paper Checklist
1. Claims
Question:
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
We emphasize the contributions and scope in the Introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: The limitation of the proposed algorithm has been discussed the supplemen- tary material.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
14

---

Justification:
There is no theoretical result.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We provide comprehensive implementation details both in main paper and in supplementary material.
Guidelines:
The answer NA means that the paper does include experiments. If the paper includes experiments, No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how reproduce that algorithm.
(b) If the contribution primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
15

---

Answer: [No]
Justification: As we promised, the data and code will be released upon the publication of our paper.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nipp) cc/
publ icl wewpddde/ooeeeaeeeeeeeeee
ss Lonooli cy) for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run reproduce the results. See the NeurIPS code and data submission guidelines (ht
/ http:/scccppbcccccuuceeee
CooeeoooeSSSa Submi ss cisioooPiiyyy for more details. The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: : The experimental setup, including data splits, training and testing detailed, are provided in Method and Experiments sections.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer:
[No]
Justification: We follow the default evaluations in the HOI detection field, which doesn't t require error bars.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
16

---

The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: We provide them in implementation details of main paper and supplementary material.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't 't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification: This work conforms the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is a special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
The border impacts is provided in supplementary material.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
17

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Question:
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, scraped datasets)?
Answer: [NA]
Justification: The proposed method uses pre-trained models. This proposed methods is safe under the safeguards of adopted pre-trained models.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification:
We cited the original paper that produced the code package dataset.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
assets are released, the license, copyright information, and terms of use the package should be provided. For popular datasets, paperssww.. www.osocittcceeeoeo com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
18

---

If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: There is new assets released in this work.
Guidelines:
The answer NA means that the paper does not release new assets. Researchers should communicate the details of the dataset/cooe/mmoel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question:
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: There is no research with human subjects in this work.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
19

---

Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that if the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality the research, declaration is not required.
Answer: [NA]
Justification: LLM does not impact the core methodology, scientific rigorousness, origi- nality the research.
Guidelines:
The answer NA means that the method development in this research does involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy (https: http://eenip..e.eee. httprriis..cccoooeeeneeeee///e/e 2025/ LLM) for what should should not described.
2025/
20


Reviews:
[{"review_id": "VTKImcraov", "reviewer": "Reviewer_cgtz", "summary": "This paper proposes a new model for Human-Object Interaction (HOI) detection. Specifically, the authors propose a two-branch model in which the first branch includes three modules: Context-aware Instruction Generator (CIG), Interest Token Selector (ITS), and Multi-Modal Reasoning (MMR). CIG works by leveraging an off-the-shelf object detector (DETR) to detect both humans and the objects, use the detected HOI region to compute spatial and appearance features, and then fuse the features with the textual instructions (e.g. \"Reason the interaction between human and objects\") to produce a \"Context-aware instruction\". Then, ITS takes the output from CIG to predict which local image split best correlates with the HOI interactions and use that to guide the local image features to be fed into the MMR module (i.e. the LLM). The MMR will then decode the text outputs describing the human-object interactions. \n\nThe second branch is somewhat standard in that it takes in the input image, encode the image with the vision encoder, then it has a Visual Interaction Decoder that takes in the appearance features from the detected HOI region as well as the global vision features to predict the HOI label out from a vocabulary. \n\nThe authors have tested the proposed method on two prominent HOI datasets HICO-DET and V-COCO, and demonstrated that the proposed model has consistently improved over prior state-of-the-arts.", "questions": "- Have you experimented with full finetuning for the language model? Wonder how that works compared to LORA tuning adopted in the paper\n- L152, why freeze the image encoder? As several papers (e.g. Cambrian paper by Tong et al) suggested the benefits of joint tuning for image encoders. \n- For training the Multi-modal Reasoning module, have you considered directly task it to predict labels in L_t's taxonomy? What's the benefits of computing textual similarity between L_t and L_p?", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "+ The paper is mostly well-written. Despite the many modules and branches in the proposed model, the authors have generally done a good job describing the proposed method. As for the description of the experiments, the texts are also clear and to the point. \n+ The proposed method, despite the somewhat complicated design, overall looks reasonable. I did not catch any obvious technical errors. \n+ The proposed method has been shown to consistently improve over prior state-of-the-arts on Human-Object-Interaction, across two prominent benchmarks.\n+ The ablation experiments presented in the Experiments section dissect different components of the proposed method, which is particularly important given the multi-stage nature of the method. \n- Table 1, in addition to the R50 + ViT-L, the proposed method also leverages a ViT for visual interaction decoder, and a LLM which should be taken into account when comparing different models. Can you provide the full parameter counts of the model including all modules? It would be good to have full parameter size comparison in Table 1. \n- The proposed model consists of a complicated pipeline involving several models (DETR for object detection, InternVL2 image encoder and LLM, ViT for visual interaction decoder), it's important to establish a comparison to end-to-end prediction using recent VLM models with decent sizes (e.g. Qwen2.5-VL models of 3B/7B sizes). Note this is different to providing justification to the effectiveness of each individual component as is done in the current ablation studies. \n- Regarding the Interest Token Selector (ITS), an important ablation that's missing is to compare to just selecting the local split that overlaps the most with the detected HOI regions.\n- Eq 4, what's the shape of T_l? It's not clear from the texts how \\hat{T}_{in} and T_l are concatenated\n- Eq 8, we need an ablation to test the accuracy of just using the LLM outputs (S2)", "was_revised": true, "final_justification": "I thank the authors for their efforts in the rebuttal. The authors have addressed most of my concerns and questions, I'm overall happy with the quality of the paper and the rebuttal consolidated my position view on the contribution of this work. Thus I will keep my rating to advocate accept the paper.", "rebuttal": "Dear reviewer cgtz,\n\nFirst and foremost, we sincerely appreciate your thorough review and valuable feedback. Your recognition of our method is greatly appreciated, e.g., \"well-written\", \"reasonable design\" and \"ablation experiments given the multi-stage nature of the method\".\n\nBelow, we provide detailed responses to each of your questions (note: the metrics for HICO-DET in the following table are reported under the \"default full setting\", while those for V-COCO are  $AP_{role}^{\\\\#1}$; the references maintain their original numbering from the main paper):\n\n--------------------------------------------------------------------------------------------------------------\n\n**Q1: Full parameter size comparison in Table 1.**\n\n**A1:** For a clearer comparison, we add the \"Large Foundation Model” (LFM) and the “Full Parameters”(FP) and “Trainable Parameters” (TP) in Table 1, as shown below. Compared to existing methods, InstructHOI achieves substantial performance gains with comparable number of trainable parameters (42.3M) and full parameters (1.13B). \n\n| Method        | LFM             |    FP(B)    |    TP(M)   |   HICO-DET   |   V-COCO    |\n|---------------|-----------------|:----------:|:-------------------------:|:--------------:|:------------------------------:|\n| DiffusionHOI[65]  | Stable Diffusion\\&CLIP |  1.02   |  27.6   |    38.12     |    66.8     |\n| UniHOI-l[22]      | BLIP-2          |  1.14   |  52.3   |    40.95     |    68.1     |\n| MP-HOI[43]        | Stable Diffusion\\&CLIP|  1.05   |  41.5   |    44.53     |    66.2     |\n| SICHOI[23]        | ChatGPT\\&BLIP        |    -     |    -     |    45.04     |    71.1     |\n| InstructHOI  | InternVL2\\&CLIP       |  1.13   |  42.3   |    **47.68**     |    **72.4**     |\n\nThank you for your valuable suggestions, we will revise Table 1 in future versions for clearer comparison.\n\n--------------------------------------------------------------------------------------------------------------\n\n**Q2: End-to-end prediction using VLM models.**\n\n**A2:** We develope an end-to-end HOI detector using Qwen2.5VL-7B for comparison. As shown in the table below, the pretrained **Qwen2.5VL-7B** achieves a **24.72 mAP** on HICO-DET (with the default full setting), while the fine-tuned **Qwen2.5VL-7B-SFT** (fine-tuned on the HOI dataset) improves this performance to **32.64 mAP**. However, both variants are significantly outperformed by InstructHOI (only equipped with a 1B-parameter MLLM). This performance gap can primarily be attributed to the limited grounding precision and reduced interaction-prediction capabilities of MLLMs models within ambiguous and complex scenarios.\n\n| Model               | HICO-DET |\n|---------------------|:----------:|\n| Qwen2.5VL-7B        | 24.72    |\n| Qwen2.5VL-7B-SFT    | 32.64    |\n| InstructHOI         | **45.95**    |\n\n--------------------------------------------------------------------------------------------------------------\n\n**Q3: Select the local split with the detected HOI regions.**\n\n**A3:** We conduct an experiment in which we select the local split that overlaps the most with the detected HOI regions (Overlap-based Strategy). As shown in the table below, this overlap-based strategy results in a 1.22 mAP performance drop compared to the Interest Token Selector (ITS). Directly using the detected HOI region as the interaction region may **neglect discriminative features and introduce irrelevant information**. In contrast, ITS evaluates the interaction relevance of local images for each human-object pair based on reasoning instructions, allowing it to more effectively select the region most closely associated with the interaction.\n\n| Token Selection Strategy | HICO-DET |\n|--------------------------|:----------:|\n| No Selection             | 45.90    |\n| Overlap-based Strategy      | 46.46    |\n| Interest Token Selector  |  **47.68**    |\n\n--------------------------------------------------------------------------------------------------------------\n\n**Q4: The shape of $T_l$.**\n\n**A4:** In Eq. (4), $ T_l $ denotes the discretized local image token sequence generated with the dynamic image encoding strategy of InternVL2, with a shape of $ N_l \\times N_s \\times d $ (where $ N_l $ represents the number of local images and $ N_s $ represents the number of tokens per local image). $ \\overline{{T}}_{in} $ refers to the representative instruction tokens, with a shape of $ N_p \\times n \\times d $. Consequently, the concatenated feature has a shape of $ N_p \\times N_l \\times (N_s + n) \\times d $.\n\n--------------------------------------------------------------------------------------------------------------\n\n**Q5: Ablation experiment using the LLM outputs (S2) in Eq. (8).**\n\n**A5:** We conduct an ablation study by using only the LLM’s outputs (S2) for interaction prediction (in E.q.(8)) on the HICO-DET. As shown in the table below, this configuration suffers a 5.09 mAP decline in the default full setting compared to the combined \"S1 + S2\" approach. In contrast, integrating the visual interaction decoder (S1) with MLLM reasoning (S2) for HOI detection effectively preserves the generalization ability of the MLLM while mitigating hallucinations, leading to a noticeable improvement in overall performance.\n\n| Interaction Score | HICO-DET |\n|-------------------|:----------:|\n| S2                | 40.86    |\n| S1+S2             | **45.95**    |\n\n--------------------------------------------------------------------------------------------------------------\n\n**Q6: Full finetuning vs LoRA finetuning.**\n\n**A6:** We conduct a comparative analysis of full fine-tuning versus LoRA fine-tuning strategies for HOI detection (see the table below). The results show that LoRA fine-tuning outperforms full fine-tuning, yielding 45.95 mAP versus 43.54 mAP. As a **lightweight adaptation method**, LoRA more effectively **preserves LFM’s knowledge learned in pre-trained stage and maintains the  generalization capability**, thus enhaning the interaction reasoning capabilities in complex scenarios.\n\n| Finetuning Strategy | HICO-DET |\n|---------------------|:----------:|\n| Full Finetuning     | 43.54    |\n| LoRA Finetuning     | **45.95**    |\n\n--------------------------------------------------------------------------------------------------------------\n\n**Q7: Freeze or unfreeze the image encoder.**\n\n**A7:** We conduct an ablation experiment by jointly training the LLM, the image encoder and the V-L MLP. The experimental results, presented in the table below, show that training only the LLM yields a relative advantage, with a 1.33mAP improvement in mAP. In the InstructHOI, we fine-tune the language model of LFMs to better understand the reconstructed HOI-specific token sequences. For relatively small-scale datasets and single-domain fine-tuning, freezing the image encoder while training only the language model may help prevent catastrophic forgetting and preserve the generalization ability of pre-trained LFMs.\n\n| Training Strategy | HICO-DET |\n|-------------------|:----------:|\n| Unfreeze          | 44.62    |\n| Freeze            | **45.95**    |\n\n--------------------------------------------------------------------------------------------------------------\n\n**Q8: Task interaction reasoning to predict labels in L_t's taxonomy?**\n\n**A8:** Computing textual similarity provides a more flexible approach to open-world interaction recognition. For instance, detecting unseen or novel interaction categories can be achieved through cosine similarity matching between the generated texts from MLLMs and the textual labels of unseen or novel categories. In contrast, directly assigning interaction reasoning to predict labels within the taxonomy of L_t's requires the design of specific classification heads for predefined interaction categories. However, when new interaction categories are introduced, the model structure needs to be modified and retrained. \n\n--------------------------------------------------------------------------------------------------------------\n\nOverall, we sincerely appreciate your valuable and constructive suggestions, and hope our responses could addresse your concerns. Please don’t hesitate to share any further feedback or questions.\n\nIn addition, we will incorporate the suggested revisions into the future version of the paper, e.g., the parameter count in Table 1, end-to-end prediction using VLM, local image token filtering with overlap-based strategy, a clearer explanation of E.q. (4), the ablation experiment on S1 and S2 in E.q. (8), and so on. These revisions will significantly enhance the quality of the paper, enabling readers to gain a more comprehensive understanding and evaluation of our work."}, {"review_id": "QHmQ5fgzza", "reviewer": "Reviewer_hkno", "summary": "The paper introduces InstructHOI, a novel architecture that leverages context-aware instructions to guide large foundation models (LFMs) in performing multimodal reasoning for human-object interaction (HOI) detection. A Context-aware Instruction Generator (CIG) is developed to fuse visual interactive context (appearance/spatial features) with linguistic instructions, providing pair-level reasoning guidance that outperforms language-only instructions. An Interest Token Selector (ITS) dynamically filters and reorganizes image tokens based on context-aware instructions, effectively aligning the reasoning process with relevant interaction regions while reducing noise. Extensive experiments demonstrate state-of-the-art results on HICO-DET and V-COCO datasets.", "questions": "Please see [Weaknesses] section.", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "[Strengths]\n\n1. The paper is well-written and easy to follow.\n\n2. Clear motivation and novel architecture design. The paper identifies the limitation of prior LFM-based HOI approaches, which focus mostly on static prompt-based knowledge transfer and largely ignore reasoning capabilities. And it introduces InstructHOI to leverage context-aware instructions for guiding large foundation models (LFMs) in human-object interaction (HOI) detection.\n\n3. The proposed CIG combines visual-human-object context (appearance and spatial features) with linguistic instructions for more contextually aware reasoning, which is more advanced than conventional prompts. The Interest Token Selector (ITS) dynamically filters and reorganizes image tokens based on interaction relevance, effectively focusing the model's attention on crucial human-object regions. This spatial adaptation mechanism significantly improves reasoning accuracy.\n\n4. SoTA performance. Extensive experiments demonstrate state-of-the-art results on HICO-DET and V-COCO datasets.\n\n[Weaknesses]\n\n1. Lack of efficiency metrics. The paper only compares the backbone networks of different LFM-based HOI methods (Table 1 in the main paper) and trainable parameters (Table 1 in the supplementary materials), while training and inference costs are also important. It is suggested that the authors provide in Table 1 of the main text the specific LFM used by different methods and their corresponding parameter counts, the amount of training data used, average inference time, FLOPs, and other metrics.\n\n2. Lack of visualizations. The proposed Interest Token Selector (ITS) module is interesting, and visualizations could more clearly validate its effectiveness. The authors only provide one visualization result of ITS in the supplementary materials, which raises questions about whether this is coincidental or whether there are bad cases. It is suggested that the authors provide more visualization results.\n\n3. Applying LFMs to the HOI domain is a challenging topic. This paper collected 5 existing image-only HOI datasets for HOI-domain fine-tuning, which is used for subsequent Multi-Modal Reasoning (MMR). Given the powerful visual understanding capabilities of multimodal large models, is LoRA fine-tuning alone sufficient to perform excellently on HOI datasets? It is suggested that the authors provide this result as an additional baseline.", "was_revised": true, "final_justification": "Thanks for the author's detailed response. Most of my concerns have been addressed. Given that my initial score was already leaning positive, I will keep the score unchanged.", "rebuttal": "Dear reviewer hkno,\n\nFirst and foremost, we extend our deepest gratitude for your thorough review and insightful feedback, as well as your positive recognition of our method, e.g.,  \"clear motivation\", \"novel architecture design\" and \"SoTA performance\".\n\nHerein, we provide a detailed response to each of your concerns (note: the metrics for HICO-DET in the following table are reported under the \"default full setting\", while those for V-COCO are  $AP_{role}^{\\\\#1}$;  the references maintain their original numbering from the main paper):\n\n--------------------------------------------------------------------------------------------------------------\n\n**Q1: Efficiency metrics in Table 1.**\n\n**A1:** Following prior work, we add the most commonly used metrics (\"Large Foundation Model\" (LFM), \"Training Data\" (TD), \"Full Parameters\" (FP), \"Trainable Parameters\" (TP), \"Inference Time\" (IT), and \"Training Time\" (TT))  in Table 1, to provide a clearer efficiency comparison, as shown below. All measurements are obtained using Tesla A800 GPUs (or GPUs with comparable performance).\n\n| Method       | LFM           | TD(K)    | FP(B)      | TP(M)     | IT(ms)    | TT(h)        | HICO-DET | V-COCO |\n|--------------|---------------|:--------:|:---------:|:--------:|:-------:|-----------|:----------:|:--------:|\n| DiffusionHOI[65] | Stable Diffusion\\&CLIP | 48    | 1.02   | 27.6  | 105 | 17.2     | 38.12    | 66.8   |\n| UniHOI-l[22]       | BLIP-2        | 48    | 1.14   | 52.3  | 82  | 18.7  | 40.95    | 68.1   |\n| MP-HOI[43]       | Stable Diffusion\\&CLIP | 286   | 1.05   | 41.5  | 196 | 23.8     | 44.53    | 66.2   |\n| SICHOI[23]       | ChatGPT\\&BLIP       | 48    | -       | -      | -     | -         | 45.04    | 71.1   |\n| InstructHOI  | InternVL2\\&CLIP     | 140   | 1.13  | 42.3  | 267 | (7.2+11.1)| **47.68**    | **72.4**   |\n\nCompared to existing methods, InstructHOI demonstrates notable performance improvements while maintaining comparable model parameters (1.13B full parameters and 42.3M trainable parameters) and computational costs (267ms inference time and a total training time of 18.3 hours). Specifically, the 7.2 hour training time refers to the HOI-SFT time for the MLLMs, while the 11.1 hours represents the second-stage training time for the full model’s component.\n\n--------------------------------------------------------------------------------------------------------------\n\n**Q2: Lack of visualizations.**\n\n**A2:**  Thank you for pointing this out. Due to rebuttal guidelines, we are unable to upload images here. As for Interest Token Selector (ITS) module, we ensure that additional visualizations will be introduced in the revised version to more clearly validate  the effectiveness. Other visualizations—covering Interaction Prediction in Two Branches, Failure Case Analysis, and In-the-Wild HOI Detection—have been provided in the supplementary materials.\n\nAdditionally, to quantitatively validate the effectiveness of ITS module, we have conducted an ablation experiment in Table 3 (main paper). The results demonstrate a 0.97 mAP increase on HICO-DET (a relative gain of 2.2 %) and a 0.6 mAP increase on V-COCO (a relative gain of 0.9 %).\n\n--------------------------------------------------------------------------------------------------------------\n\n**Q3: Directly use LoRA-finetuned MLLM for interaction prediction.**\n\n**A3:**  We introduce an additional baseline in which the LoRA-finetuned MLLM is directly employed for HOI interaction recognition, referred to as Multi-Modal Reasoning (MMR). As shown in the table below, MMR alone results in a 5.09 mAP drop on the HICO-DET compared to InstructHOI. The standalone MMR may be prone to hallucinations in complex and ambiguous scenarios. In contrast, integrating the visual interaction decoder (S1) with MMR (S2) for HOI detection effectively maintains the generalization capability of the MLLM while mitigating hallucinations, leading to a significant improvement in overall performance.\n\n| Prediction Method | HICO-DET |\n|-------------------|:----------:|\n| MMR               | 40.86    |\n| InstructHOI       | **45.95**  |\n\n--------------------------------------------------------------------------------------------------------------\n\nWe sincerely appreciate your valuable and constructive suggestions, and we hope that our response could addresses your concerns. Please feel free to provide any further feedback if you have additional questions. \n\nIn addition, we will incorporate your suggested revisions into the future version of the paper, including: adding efficiency metrics in Table 1, providing visual results for ITS, and including additional baseline experiments on directly using LoRA-finetuned MLLMs for interaction prediction. These revisions will significantly enhance the quality of the paper, allowing readers to gain a more comprehensive understanding and evaluation of our work."}, {"review_id": "0vg8KqQmIO", "reviewer": "Reviewer_oDfg", "summary": "In this paper, the authors address the reasoning capabilities of the foundation model in HOI task. Different from previous methods which directly using the pretrained LFM. A large HOI image-text pair dataset is used to fine-tune the LFM by LoRA at first. Then, the LFM is adopted to predict the interaction of the H-O pair with the resoning token from the proposed Context-Aware Instruction Genrator (CIG) and Interest Token Selector (ITS). The CIG translates the HOI visual embedding into linguistic space and the ITS selects the interaction-relevant image tokens from the visual features. The extensive experiments demonstrate the effectiveness of the proposed method in improving the performance of HOI detection.", "questions": "1. In Line 235, the cosine similarity operation is calculated between two sets of texts. What model is used to extract the text embeddings? E.g. CLIP, BLIP, or other text encoders.\n\n2. With the object detector and the LFM, the model already has the ability to predict the interaction probability $S_2$ of the H-O pair. The LFM also uses both of the appearance and the H-O pair embeddings, is it necessary to use the Visual Interaction Decoder?\n\n3. The interaction-relevance matrix $M_ir$ in ITS used to filter the image token of local features, however, building the connection between the visual features from the object detector and the LFM is implicit, the matrix may not be accurate to focus on the interaction regions. Why not directly use the predicted bounding boxes to filter the image token?\n\n4. As mentioned in weaknesses, the 140K dataset used to fine-tune the LFM may contain the HOI categories in zero-shot settings, are these related categories' data removed from the fine-tuning dataset? If not, the comparison with the previous methods on the zero-shot setting is not fair.", "limitations": "Yes.", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 4, "strengths_and_weaknesses": "Strengths:\n\nThe paper proposed a novel method to enhance the reasoning capabilities of the foundation model in HOI task and achieved state-of-the-art performance on the HOI benchmark datasets. The manuscript is clearly presented with well-structured sections and figures. This paper addresses a significant problem in the HOI task, the fine-tuning strategy of the HOI model incorporating with the LFM is innovative and is important for the community.\n\nWeaknesses:\n\nAs the LFM is fine-tuned on a 140K HOI image-text pair dataset, which contains whole HOI categories, the comparison with the previous methods on the zero-shot setting is not fair. While using the pretrained LFM, the improvement is limited. The proposed method enhances the reasoning capabilities during the fine-tuning stage of the LFM, however, the prediction is still based on the text labels and an interaction decoder is still needed to predict the interaction probability, thus, the interaction representation capabilities of the LFM is still limited.", "was_revised": true, "final_justification": "The paper proposed a novel method to enhance the reasoning capabilities of the foundation model in HOI task and achieved state-of-the-art performance on the HOI benchmark datasets. The manuscript is clearly presented with well-structured sections and figures. This paper addresses a significant problem in the HOI task, the fine-tuning strategy of the HOI model incorporating with the LFM is innovative and is important for the community.\n\nThank you to the authors for the detailed and thorough response. Your reply has sufficiently addressed my concerns. Given the inspiration this work may provide for future research, I am willing to increase my score.", "rebuttal": "Dear reviewer oDfg,\n\nFirst and foremost, we would like to express our sincere gratitude for your detailed review and valuable feedback. We truly appreciate your recognition of our method, e.g., \"novel method\", \"addresse a significant problem in the HOI task\" and \"innovative and important for the community\".\n\nBelow, we provide a comprehensive response to each of your concerns (please note: the metrics for HICO-DET in the table below are reported under the \"default full setting,\" while those for V-COCO are $AP_{role}^{\\\\#1}$; the references maintain their original numbering from the main paper):\n\n--------------------------------------------------------------------------------------------------------------\n\n**Q1: Text embeddings model.**\n\n**A1:** Following prior work, we employ CLIP’s text encoder to embed both the generated text and the textual labels, and then compute their similarity for interaction prediction.\n\n--------------------------------------------------------------------------------------------------------------\n\n**Q2: Necessity of Visual Interaction Decoder.**\n\n**A2:** We conduct an ablation study by using only the LLM’s outputs (S2) for interaction prediction on the HICO-DET. As shown in the table below, relying solely on S2 results in a 5.09 mAP drop in the default full setting compared to the combined \"S1 + S2\" approach (where S1 represents interaction scores generated by the Visual Interaction Decoder). These experimental results underscore the critical role of the Visual Interaction Decoder: by fusing the visual interaction encoder (S1) with MLLM reasoning (S2), InstructHOI could effectively preserves the generalization ability of the MLLM and mitigate hallucinations, leading to superior overall performance.\n\n| Interaction Score | HICO-DET |\n|-----------------|:--------:|\n| S2                |  40.86   |\n| S1 + S2           |  **45.95**   |\n\n--------------------------------------------------------------------------------------------------------------\n\n**Q3: Directly use the predicted bounding boxes to filter the image token.**\n\n**A3:**  We conducted an experiment in which we directly used the predicted bounding boxes to filter the image tokens (region-overlap strategy). As shown in the table below, this region-overlap strategy results in a 1.22 mAP performance drop compared to the Interest Token Selector (ITS). **Directly using the detected HOI region as the interaction region may neglect discriminative features and introduce irrelevant information**.  \n\nAlthough the connection between the visual features extracted from the object detector and the LFM is implicit, the instruction projector (Sec. 3.3) could aligns these two visual feature spaces via feature mapping. ITS evaluates the interaction relevance of each local patch for each human-object pair, thereby more effectively selecting the interaction-related region.\n\n| Token Selection Strategy  | HICO-DET |\n|---------------------------|:---------:|\n| No Selection              |    45.90 |\n| Region-overlap Strategy       |    46.46 |\n| Interest Token Selector   |    **47.68** |\n\n--------------------------------------------------------------------------------------------------------------\n\n**Q4: Removal of Data from Related Categories.**\n\n**A4:**  To ensure a fair zero-shot evaluation, we excluded all training data related to the held-out HOIs from the fine-tuning dataset, thereby preventing any potential data leakage. For instance, when \"talk on a cell phone\" (in HICO-DET) is considered an unseen HOI, we also exclude related HOI categories from the fine-tuning dataset, e.g., \"phone\" in SWIG-HOI and \"talk on phone\" in V-COCO.\n\n--------------------------------------------------------------------------------------------------------------\n\nWe truly value your insightful and constructive suggestions, and we hope our response could addresse your concerns. If you have any further questions or feedback, please do not hesitate to share them with us.\n\nIn the revised version of the paper, we will include a clearer explanation of the text embedding model, ablation experiments on the Visual Interaction Decoder, experiments on local image token filtering with detected HOI regions, and a more detailed explanation of data removal."}, {"review_id": "TQFLMG2s4s", "reviewer": "Reviewer_jzDg", "summary": "This paper introduces a human object interaction detection framework that utilizes the large foundation models to enhance multi-modal reasoning for HOI detection. The authors propose to fine-tune the LFM on large-scale HOI-domain data to bridge the gap between pretrained knowledge and HOI detection. The authors propose a Context-aware Instruction Generator to guide interaction reasoning and an Interest Token Selector to select important image tokens. The experimental results on two HOI benchmarks demonstrate the effectiveness of the proposed methods.", "questions": "- I would expect an ablation study on the HOI-domain fine-tuning data choice, such as only fine-tuning the LFM on HICO-DET. For the LFM choice, the authors could provide an ablation study on this issue or explain the rationale behind the choice of InternVL2.\n- Following the previous question, could the authors clarify that the *Backbone* in Table 1 refers to which part of the proposed method?\n- Could authors provide an inference and training cost comparison with baselines?", "limitations": "More ablation is expected to verify the core contribution of this work.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "# Strengths\n- This paper is well-written and easy to follow.\n- The proposed framework is well-motivated and reasonable.\n- The performance is good on popular HOI benchmarks.\n\n# Weaknesses\n- Major performance improvements come from the Multi-Modal Reasoning, which includes training on data from five HOI datasets. My concern is whether the performance gains come from additional training data or a stronger multi-modal LFM (InternVL2). These may cause unfair comparison with other baselines. \n- The necessity of Spatial Context in CIG. From the results reported in Table 4, the gains from SC are quite limited, and the design of SC requires an offline object detector model, which may introduce additional noise.\n- The frameworks introduce a lot of additional modules compared with the baselines. A comparison of inference and training cost is expected to reveal the trade-off between efficiency and effectiveness of proposed methods.", "was_revised": true, "final_justification": "After reading the rebuttal and other reviewers' comments, my concerns are addressed. So I will maintain my positive score.", "rebuttal": "Dear reviewer jzDg,\n\nWe gratefully acknowledge your comprehensive review, insightful critiques, and affirming evaluation of our approach, e.g.,  \"well-motivated\", \"reasonable\" and \"easy to follow\".\n\nBelow, we provide a detailed response to each of the points you raised (note: the metrics for HICO-DET in the following table are reported under the \"default full setting\", while those for V-COCO are  $AP_{role}^{\\\\#1}$; the references maintain their original numbering from the main paper):\n\n--------------------------------------------------------------------------------------------------------------\n\n**Q1:  Concern about additional training data or a stronger multi-modal LFM (InternVL2).**\n\n**A1:** **For finetuning data**, we have reported the results of ablation study in the supplementary materials (Table 2). As shown in the following table, the“pretrained\" indicates pretrained LFM without fine-tuning; “fine-tune$^{†}$\" indicates fine-tuning on a specific training dataset (i.e., HICO-DET or V-COCO); “fine-tune$^{‡}$\" indicates fine-tuning on the collected dataset. Notably, even when fine-tuned exclusively on HICO-DET or V-COCO (fine-tune$^{†}$), our model still achieves SOTA performance. Moreover, the “fine-tune$^{‡}$” strategy yields an additional gain of 0.59 mAP on the HICO-DET benchmark and 0.40 mAP on the V-COCO, demonstrating that finetuning on the larger dataset could further enhance InstructHOI's generalization ability.\n\n| Strategy     | HICO-DET | V-COCO |\n|--------------|:----------:|:--------:|\n| pretrained   | 43.68    | 69.4   |\n|fine-tune$^{†}$   | 45.36    | 70.4   |\n| fine-tune$^{‡}$   | **45.95**    | **70.8**   |\n\n\n**For LFM choice**, we have reported the results of ablation study on LFMs in the supplementary materials (Table 1). As shown in the following table, we explore variants of InstructHOI that leverage different LFMs, including LLaVA-OV and InternVL2. Furthermore, for each LFM variant, we assess the impact of different foundation model scales. The experimental results demonstrate that InstructHOI achieves SOTA performance with both LLaVA-OV and InternVL2. InternVL2 demonstrates slightly better performance, which could be attributed to its strong foundational and visual perception capabilities. Additionally, as the scale of the underlying LFM increases, InstructHOI’s performance further improves, highlighting the enhanced reasoning capabilities at larger model scales.\n\n| Methods - *LFMs*           | Foundation Models           | V-COCO |\n|--------------------------|-----------------------------|:--------:|\n| InstructHOI-$_{LLaVA-OV}$      | ViT-L Qwen2$_{0.5B}$           | 70.3   |\n| InstructHOI-$_{LLaVA-OV}$      | ViT-L Qwen2$_{7B}$             | **71.4**   |\n| InstructHOI-$_{InternVL2}$   | ViT-L Qwen2$_{0.5B}$            | 70.8   |\n| InstructHOI-$_{InternVL2}$    | ViT-L InternLM2$_{7B}$        | **72.1**   |\n\n\n--------------------------------------------------------------------------------------------------------------\n\n**Q2: The necessity of Spatial Context in CIG.**\n\n**A2:** The primary objective of integrating an offline object detector is not to extract Spatial Context (SC), but to achieve precise instance localization. Consistent with prior work [21], [58] demonstrating the importance of spatial relationships for interaction prediction, the experimental results in Table 4 (in the main paper) show that the introduction of SC token improves performance on the HICO-DET by 0.78 mAP (**1.8% relative improvement**). Moreover, when the LFM model is scaled up to 7B parameters (InternVL2-7b), the gains of SC further increase to 1.42mAP (**3.1% relative improvement**) on the HICO-DET. LFMs with greater foundational capacity demonstrate an enhanced ability to comprehend SC token and leverage it for interaction reasoning.\n\n--------------------------------------------------------------------------------------------------------------\n\n**Q3: Clarify that the Backbone in Table 1.**\n\n**A3:**  For VLM-based HOI detectors, the term \"backbone\" (e.g., R101+ViT-L) refers to the combination of the basic feature extractor (e.g., R101 for object detection) and the visual encoder backbone of the large foundation model (e.g., ViT-L). Additionally, to provide a clearer perspective, we also add the specific Large Foundation Model (LFMs) of different methods in Table 1, as following:\n\n| Method         | Backbone        | LFM              |\n|----------------|-----------------|------------------|\n| DiffusionHOI[65] | VQGAN+ViT-L           | Stable Diffusion\\&CLIP |\n| UniHOI-l[22]    | R101+ViT-L      | BLIP-2           |\n| MP-HOI[43]      | Swin-L+ViT      | Stable Diffusion\\&CLIP |\n| SICHOI[23]      | R101+ViT-L      | ChatGPT\\&BLIP          |\n| InstructHOI     | R101+ViT-L      | InternVL2\\&CLIP        |\n\n--------------------------------------------------------------------------------------------------------------\n\n**Q4: Inference and training cost.**\n\n**A4:** We evaluated the inference time of our model in comparison to high-performance HOI detectors, utilizing Tesla A800 GPU (or GPU with comparable performance). The results are presented in the table below, compared to existing methods, InstructHOI demonstrates notable performance improvements while maintaining comparable computational costs.\n\n| Method             | Inference time (ms) | HICO-DET |\n|--------------------|:---------------------:|:----------:|\n| ADA-CM[66]         | 286                | 38.40    |\n| DiffusionHOI[65]   | 105                 | 38.12    |\n| UniHOI[43]         | 82                  | 40.95    |\n| MP-HOI[23]         | 186                 | 44.53    |\n| InstructHOI        | 267                 | **47.68**    |\n\nWe evaluated the training time of our model in comparison to high-performance HOI detectors, utilizing 8 Tesla A800 GPUs (or GPUs with comparable performance). The results are presented in the table below, specifically, the 7.2 hour training time refers to the HOI-SFT time for the MLLMs, while the 11.1 hours represents the second-stage training time for the full model’s component.\n\n| Method             | Training time (h) | HICO-DET |\n|--------------------|:----------------------:|:----------:|\n| HOICLIP[44]        | 26.4                 | 34.69    |\n| DiffusionHOI[65]   | 17.2                 | 38.12    |\n| UniHOI[43]         | 18.7                 | 40.95    |\n| MP-HOI[23]         | 23.8                 | 44.53    |\n| InstructHOI        | 7.2+11.1             | **47.68**    |\n\n--------------------------------------------------------------------------------------------------------------\n\nWe greatly appreciate your insightful and constructive feedback, and we hope our response could addresses your concerns. If you have any further questions, please don’t hesitate to share them with us.\n\nIn the revised version of the paper. we will provide a clearer explanation of the backbone component and the role of the spatial context token, as well as include more details on the inference and training time."}]

Author rebuttal:

Meta-review: The paper presents a novel 2 branch system for the task of HOI detection. The first branch is a more standard vision encoder -> HOI decoder, which has been shown many times to be effective. The second branch is specially designed for the HOI task and imbues the model the ability to take advantage of context-aware instructions that encourage better reasoning about the interactions taking place in the image.  This idea makes a lot of sense and the proposed system leverages current llm/vlm tools well to achieve this better reasoning. Importantly the authors show that using this method they can train on a large corpus of datasets (combining HOI datasets) to show generalization as well as just the specific train split for the test set to show fairness to other methods. The authors are careful about removing HOIs that are present in any test set to make sure their method is reasoning across unseen interaction pairs not just memorizing.

The combination of logical and novel system with strong, consistent improvements to SOTA warrant selection as an spotlight. The method could be a standard baseline for HOI reasoning going forward. 

All reviewers agree the paper should be accepted and the AC agrees. The paper is a strong candidate for an spotlight.
Decision: Accept (spotlight)