SlowGuess/NeurIPS_2023_OCR huggingface dataset:
where  $x$  refers to one human-object pair that is potential to have interactions. In first-order logic, the semantics of variables (e.g.,  $x$ ) is usually referred to predicates (e.g., launch(x), above(x)). Eq. 7
states that, for instance, if the  $v$  is launch, and  $p$  is above, then in addition to interactions composed of actions apart from launch, human-launch-boat should be included in  $\{h_1, \dots, h_N\}$  as well. Similarly, given the object category  $o$  and position relationship  $p$ , we shall have:
$$
\forall x (o (x) \wedge p (x) \rightarrow \neg h _ {1} (x) \wedge \neg h _ {2} (x) \wedge \dots \wedge \neg h _ {N} (x)). \tag {8}
$$
With Eq.7 and Eq.8, both affordances and proxemics properties, and the combination relationship between them are clearly stated. Next we investigate how to convert the above logical symbols into differentiable operation. Specifically, logical connectives (e.g.,  $\rightarrow$ ,  $\neg$ ,  $\lor$ ,  $\land$ ) defined on discrete Boolean variables are grounded to functions on continuous variables using product logic [108]:
$$
\psi \rightarrow \phi = 1 - \psi + \psi \cdot \phi , \quad \neg \psi = 1 - \psi , \tag {9}
$$
$$
\psi \lor \phi = \psi + \phi - \psi \cdot \phi , \quad \psi \land \phi = \psi \cdot \phi .
$$
Similarly, the quantifier are implemented in a generalized-mean manner following [109]:
$$
\exists x (\psi (x)) = \left(\frac {1}{K} \sum_ {k = 1} ^ {K} \psi \left(x _ {k}\right) ^ {q}\right) ^ {\frac {1}{q}}, \tag {10}
$$
$$
\forall x (\psi (x)) = 1 - (\frac {1}{K} \sum_ {k = 1} ^ {K} (1 - \psi (x _ {k})) ^ {q}) ^ {\frac {1}{q}},
$$
Given above relaxation, we are ready to translate properties defined in first-order logical formulae into sub-symbolic numerical representations, so as to supervise the interactions  $\{h_1,\dots ,h_M\}$  predicted by the Transformer reasoner. For instance, Eq. 7 is grounded by Eq. 9 and Eq. 10 into:
$$
\mathcal {G} _ {v, p} = 1 - \frac {1}{M} \sum_ {m = 1} ^ {M} \left(\frac {1}{K} \sum_ {k = 1} ^ {K} \left(s _ {k} [ v ] \cdot s _ {k} \left[ h _ {m} \right]\right)\right), \tag {11}
$$
where  $s_k[v]$  and  $s_k[h_m]$  refer to the scores of action  $v$  and interaction  $h_m$  with respect to input sample  $x_k$ . Here  $K$  refers to the number of all training samples and we relax it to that in a mini-batch. As mentioned above, the spatial locations of human and objects are concatenated into the query which means the spatial relation is predetermined and can be effortlessly inferred from the box predictions (details provided in Supplementary Materials). Thus, we omit  $p(x)$  in Eq.11. Then, the action-position loss is defined as:  $\mathcal{L}_{v,p} = 1 - \mathcal{G}_{v,p}$ . In a similar way, we can ground Eq.8 into:
$$
\mathcal {G} _ {o, p} = 1 - \frac {1}{N} \sum_ {n = 1} ^ {N} \left(\frac {1}{K} \sum_ {k = 1} ^ {K} \left(s _ {k} [ v ] \cdot s _ {k} \left[ h _ {n} \right]\right)\right), \tag {12}
$$
where  $s_k[o]$  refers to the score of object regarding to the input sample  $x_k$ . The object-position loss is defined as:  $\mathcal{L}_{o,p} = 1 - \mathcal{G}_{o,p}$ . For  $\mathcal{G}_{v,p}$ , it scores the satisfaction of predictions to rules defined in Eq.7. For example, given a high probability of action ride (i.e., a high value of  $s_k[v]$ ) and the position relationship is above, if the probability of infeasible interactions (e.g., human-feed-fish") is also high, then  $\mathcal{G}_{v,p}$  would receive a low value so as to punish this prediction.  $\mathcal{G}_{o,p}$  is similar but it scores the satisfaction of predictions to Eq.8 with given position and objects such as horse, fish, etc. Through Eq.11 and Eq.12, we aim to achieve that, given the embeddings and locations of a group of human and object entities, along with the potential actions happened within the image, the Transformer reasoner should speculate which pair of human-object engaged in what kind of interaction, while the prediction should respect to the rules defined in Eq.7 and Eq.8.
Network Architecture. To make fair comparison with existing Transformer-based work [9-20], we adopt ResNet-50 as the backbone. The visual encoder  $\mathcal{E}$  is implemented as six Transformer encoder layers, while the three parallel human, object and action decoders  $\mathcal{D}^h$ ,  $\mathcal{D}^a$ ,  $\mathcal{D}^o$  are all constructed as three Transformer decoder layers. For the interaction decoder  $\mathcal{D}^p$ , we instantiate it with three Transformer decoder layers as well, but replacing self-attention with our proposed triplet-reasoning attention. The number of human, object, action queries  $N_h$ ,  $N_o$ ,  $N_a$  is set to 32 for efficiency, and the hidden sizes of all the modules are set to  $D = 768$ . Since the state-of-the-art work [17, 19, 63-66] usually leverages large-scale visual-linguistic pre-trained models to further enhance the detection capability, we follow this setup and adopt CLIP [67]. To improve the inference efficiency of our framework, we further follow [17] which uses guided embeddings to decode humans and objects in a single Transformer decoder, i.e., merging  $\mathcal{D}^h$  and  $\mathcal{D}^o$  into a unified one to simultaneously output both human and object predictions.
Training Objectives. LOGICHOI is jointly optimized by the HOI detection loss (i.e.,  $\mathcal{L}_{\mathrm{HOI}}$ ) and logic-induced property learning loss (i.e.,  $\mathcal{L}_{\mathrm{LOG}}$ ):
$$
\mathcal {L} = \mathcal {L} _ {\mathrm {H O I}} + \alpha \mathcal {L} _ {\mathrm {L O G}}, \quad \mathcal {L} _ {\mathrm {L O G}} = \mathcal {L} _ {v, p} + \mathcal {L} _ {o, p}. \tag {13}
$$
Here  $\alpha$  is set to 0.2 empirically. Note that  $\mathcal{L}_{\mathrm{LOG}}$  solely update the parameters of the Transformer reasoner (i.e., interaction decoder  $\mathcal{D}^p$ ) but not the entire network to prevent over-fitting. For  $\mathcal{L}_{\mathrm{HOI}}$ , it is composed of human/object (i.e., output of  $\mathcal{D}^h$  and  $\mathcal{D}^o$ , respectively) detection loss, action (i.e., output of  $\mathcal{D}^a$ ) classification loss as well as interaction (i.e., output of  $\mathcal{D}^p$ ) classification loss.
Datasets. We conduct experiments on two widely-used HOI detection benchmarks:
- V-COCO [53] is a carefully curated subset of MS-COCO [110] which contains 10,346 images (5,400 for training and 4,946 for testing). There are 263 human-object interactions annotated in this dataset in total, which are derived from 80 object categories and 29 action categories.
- HICO-DET[54] consists of 47,776 images in total, with 38,118 for training and 9,658 designated for testing. It has 80 object categories identical to those in V-COCO and 117 action categories, consequently encompassing a comprehensive collection of 600 unique human-object interactions.
Evaluation Metric. Following conventions [2, 10, 37], the mean Average Precision (mAP) is adopted for evaluation. Specifically, for V-COCO, we report the mAP scores under both scenario 1 (S1) which includes all of the 29 action categories and scenario 2 (S2) which excludes 4 human body motions without interaction to any objects. For HICO-DET, we perform evaluation across three category sets: all 600 HOI categories (Full), 138 HOI categories with less than 10 training instances (Rare), and the remaining 462 HOI categories (Non-Rare). Moreover, the mAP scores are calculated in two separate setups: i) the Default setup computing the mAP on all testing images, and ii) the Known Object setup measuring the AP for each object independently within the subset of images containing this object.
Zero-Shot HOI Detection. We follow the setup in previous work[17,21,31,106,107] to carry on zero-shot generalization experiments, resulting in four different settings: Rare First Unseen Combination (RF-UC), Non-rare First Unseen Combination (NF-UC), Unseen Verb (UV) and Unseen Object (UO) on HICO-DET. Specifically, the RF and NF strategies in the UC setting indicate selecting 120 most frequent and infrequent interaction categories for testing, respectively. In the UO setting, we choose 12 objects from 80 objects that are previously unseen in the training set, while in the UV setting, we exclude 20 verbs from a total of 117 verbs during training and only use them at testing.
Training. To ensure fair comparison with existing work[9-20], we initialize our model with weights of DETR [63] as trained:
of DETR[62] pre-trained on MS-COCO. Subsequently, we conducted training for 90 epochs using the Adam optimizer with a batch size of 16 and base learning rate  $1e^{-4}$ , on 4 GeForce RTX 3090 GPUs. The learning rate is scheduled following a "step" policy, decayed by a factor of 0.1 at the 60th epoch. In line with [10, 62, 66], the random scaling augmentation is adopted, i.e., training images are resized to a maximum size of 1333 for the long edge, and minimum size of 400 for the short edge.
Testing. For fairness, we refrain from implementing
Table 1: Comparison of zero-shot generalization with state-of-the-arts on HICO-DET [54] test. See §4.2 for details.
Method	VL Pretrain	Type	Unseen	Seen	Full
VCL [31]ECCV20	RF-UC	10.06	24.28	21.43
ATL [107]CVPR21	RF-UC	9.18	24.67	21.57
FCL [106]CVPR21	RF-UC	13.16	24.23	22.01
GEN-VLKT [17]CVPR22	✓	RF-UC	21.36	32.91	30.56
SCL [21]ECCV22	RF-UC	19.07	30.39	28.08
LOGICHOI (ours)	✓	RF-UC	25.97	34.93	33.17
VCL [31]ECCV20	NF-UC	16.22	18.52	18.06
ATL [107]CVPR21	NF-UC	18.25	18.78	18.67
FCL [106]CVPR21	NF-UC	18.66	19.55	19.37
GEN-VLKT [17]CVPR22	✓	NF-UC	25.05	23.38	23.71
SCL [21]ECCV22	NF-UC	21.73	25.00	24.34
LOGICHOI (ours)	✓	NF-UC	26.84	27.86	27.95
ATL [107]CVPR21	UO	5.05	14.69	13.08
FCL [106]CVPR21	UO	0.00	13.71	11.43
GEN-VLKT [17]CVPR22	✓	UO	10.51	28.92	25.63
LOGICHOI (ours)	✓	UO	15.67	30.42	28.23
GEN-VLKT [17]CVPR22	✓	UV	20.96	30.23	28.74
LOGICHOI (ours)	✓	UV	24.57	31.88	30.77
Table 2: Quantitative results on HICO-DET[54] test and V-COCO[53] test. See §4.3 for details.
Method	Backbone	VL Pretrain	Default	Known Object	APS1role	APS2role
Full	Rare	Non-Rare	Full	Rare	Non-Rare
iCAN [111][BMVC18]	R50	14.84	10.45	16.150	16.26	11.33	17.73	45.3	-
UnionDet [58][ECCV20]	R50	17.58	11.72	19.33	19.76	14.68	21.27	47.5	56.2
PPDM [59][CVPR20]	HG104	21.73	13.78	24.10	24.58	16.65	26.84	-	-
HOTR [10][CVPR21]	R50	23.46	16.21	25.60	-	-	-	55.2	64.4
AS-Net [9][CVPR21]	R50	28.87	24.25	30.25	31.74	27.07	33.14	53.9	-
QPIC [11][CVPR21]	R50	29.07	21.85	31.23	31.68	24.14	33.93	58.8	61.0
CDN [13][NeurIPS21]	R50	31.78	27.55	33.05	34.53	29.73	35.96	62.3	64.4
MSTR [14][CVPR22]	R50	31.17	25.31	32.92	34.02	28.83	35.57	62.0	65.2
UPT [15][CVPR22]	R50	31.66	25.94	33.36	35.05	29.27	36.77	59.0	64.5
STIP [22][CVPR22]	R50	32.22	28.15	33.43	35.29	31.43	36.45	65.1	69.7
IF-HOI [18][CVPR22]	R50	33.51	30.30	34.46	36.28	33.16	37.21	63.0	65.2
ODM [112][ECCV22]	R50-FPN	31.65	24.95	33.65	-	-	-	-	-
Iwin [113][ECCV22]	R50-FPN	32.03	27.62	34.14	35.17	28.79	35.91	60.5	-
LOGICHOI (ours)	R50	34.53	31.12	35.38	37.04	34.31	37.86	63.7	64.9
CTAN [66][CVPR22]	R50	✓	31.71	24.82	33.77	33.96	26.37	36.23	60.1
SSRT [63][CVPR22]	R50	✓	30.36	25.42	31.83	-	-	-	63.7	65.9
DOQ [65][CVPR22]	R50	✓	33.28	29.19	34.50	-	-	-	63.5	-
GEN-VLK [17][CVPR22]	R50	✓	33.75	29.25	35.10	37.80	34.76	38.71	62.4	64.4
LOGICHOI (ours)	R50	✓	35.47	32.03	36.22	38.21	35.29	39.03	64.4	65.6
any data augmentation during testing. Specifically, we first select  $K$  interactions with the highest scores and further filter them by applying NMS to retrieve the final predictions. Following the convention [11, 16, 17, 20, 66], we set  $K$  to 100.
4.2 Zero-Shot HOI Detection
The comparisons of our method against several top-leading zero-shot HOI detection models[17,21,31, 106, 107] on HICO-DET test are presented in Table 1. It can be seen that LOGICHOI outperforms all of the competitors by clear margins across four different zero-shot setups.
Unseen Combination. As seen, LOGICHOI provides a considerable performance gain against existing methods. In particular, it outperforms the top-leading GEN-VLKT[17] by  $4.61\%$  and  $1.79\%$  in terms of mAP on unseen categories for RF and NF selections, respectively. These numerical results substantiate our motivation of empowering Transformer with the reasoning ability and guide the learning in a logic-induced compositional manner, rather than solely taking Transformer as an interaction classifier.
Unseen Object. Our approach achieves dominant results under the UO setting, surpassing other competitors across all metrics. Notably, it yields unseen mAP  $15.67\%$  and overall mAP  $28.23\%$ , while the corresponding scores for the SOTA methods [17, 107] are  $5.05\%$ ,  $13.08\%$  and  $10.51\%$ ,  $25.63\%$ , presenting an improvement up to  $5.16\%$  mAP for unseen categories. This reinforces our belief that empowering Transformer with logic-guided reasoning ability is imperative and indispensable.
Unseen Verb. Our method also demonstrates superior performance in the UV setup. Concretely, it surpasses GEN-VLKT[17] by  $3.61\%$  mAP on unseen categories and achieves  $30.77\%$  overall scores.
All of the above improvement on zero-shot generalization confirms the effectiveness of our proposed Transformer reasoner which learns in a compositional manner, and is informed by affordances and proxemics knowledge to address novel challenges that was never encountered before.
4.3 Regular HOI Detection
HICO-DET. In Table 2, we present the results of LOGICHOI and other top-performing models under the normal HOI detection setup. Notably, on HICO-DET[54] test, our solution demonstrates a significant improvement over previous state-of-the-art[17], with a substantial margin of  $1.72\%$  /  $2.78\% / 1.12\%$  mAP for Full, Rare, and Non-Rare categories, under the Default setup. Moreover, in terms of Known Object, our method attains exceptional mAP scores of  $38.21\% / 35.29\% / 39.03\%$ .
Table 3: Analysis of essential components of LOGI-CHOI on HICO-DET [54]. See §4.4 for details.
TRA	LRL	Full	Rare	Non-Rare
31.87	26.14	33.29
✓	34.32	30.67	35.19
✓	33.26	29.53	34.56
✓	✓	35.47↑3.60	32.03↑5.89	36.22↑2.93
Table 4: Analysis of LRL under the zero-shot setup of unseen categories. See §4.4 for details.
Setting	RF-UC	UO	UV
TRA	24.01	13.26	23.14
+Lv,p	25.22	15.32	23.68
+L0,p	25.34	13.91	24.29
LOGICHOI	25.97↑1.96	15.67↑2.41	24.57↑1.43
Table 5: Analysis of number of decoder layer and query on HICO-DET[54] test. See §4.4 for details.
# of layers (L)	Full	Rare	Non-Rare
2	34.61	30.72	35.54
3	35.47	32.03	36.22
4	35.37	31.96	36.09
6	35.61	32.13	36.39
(a) number of the interaction decoder layer.
# of queries (N)	Full	Rare	Non-Rare
16	35.06	31.36	35.94
32	35.47	32.03	36.22
64	35.26	31.65	36.06
128	34.67	30.98	35.53
(b) number of the queries.
V-COCO. As indicated by the last two columns of Table 2, we also compare LOGICHOI with competitive models on V-COCO[53] test. Despite the relatively smaller number of images and HOI categories in this dataset, our method still yields promising results, showcasing its effectiveness. In particular, it achieves a mean mAP score of  $65.0\%$  across two scenarios.
4.4 Diagnostic Experiment
For in-depth analysis, we perform a series of ablative studies on HICO-DET[54] test.
Key Component Analysis. We first examine the efficiency of essential designs of LOGICHOI, i.e., triplet-reasoning attention (TRA) and logic-guided reasoner learning (LRL), which is summarized in Table 3. Three crucial conclusions can be drawn. First, our proposed triplet-reasoning attention leads to significant performance improvements against the baseline across all the metrics. Notably, TRA achieves  $4.53\%$  mAP improvement on Rare congeries, demonstrating the ability of our Transformer Reasoner to reason over entities and generate more feasible predictions. Second, we also observe compelling gains from incorporating logic-guided reasoner learning into the baseline, even with basic self-attention, affirming its versatility. Third, our full model LOGICHOI achieves the satisfactory performance, confirming the complementarity and effectiveness of our designs.
Logic-Guided Learning. We guide the learning of Transformer reasoner with two logic-induced properties. Table 4 reports the related scores of unseen categories under three zero-shot setups. The contributions of  $\mathcal{L}_{v,p}$  and  $\mathcal{L}_{o,p}$  are approximately equal in the RF-UC setup, since during training, all actions and objects can be seen and utilized to guide reasoning. On the other hand, under the UO and UV setups, the improvements heavily rely on  $\mathcal{L}_{v,p}$  and  $\mathcal{L}_{o,p}$  respectively, while the other one brings minor enhancements. Finally, the combination of them leads to LogICHOI, the new state-of-the-art.
Number of Decoder Layer. We further examine the effect of the number of Transformer decoder layer used in  $\mathcal{D}^p$ . As shown in Table 5a, Logi-CHOI achieves similar performance when  $L$  is larger than 2. For efficiency, we set  $L = 3$  which is the smallest among existing work[9-20].
Number of Query. Next we probe the impact of the number of query for human, object and action in Table 5b. Note that the number of these three kind of queries is identical and we refer it to  $N$ . The best performance is obtained at  $N = 32$  and more queries lead to inferior performance.
Table 6: Comparison of parameters and running efficiency.
Method	Backbone	Params	FLOPs	FPS
Two-stages Detectors:
iCAN [111][BMVC18]	R50	39.8	-	5.99
DRG [30][ECCV20]	R50-FPN	46.1	-	6.05
SCG [36][ICCV21]	R50-FPN	53.9	-	7.13
STIP [22][CVPR22]	R50	50.4	-	6.78
One-stages Detectors:
PPDM [59][CVPR20]	HG104	194.9	-	17.14
HOTR [10][CVPR21]	R50	51.2	90.78	15.18
HOITrans [12][CVPR21]	R50	41.4	87.69	18.29
AS-Net [9][CVPR21]	R50	52.5	87.86
QPIC [11][CVPR21]	R50	41.9	88.87	16.79
CDN-S [13][NeurIPS21]	R50	42.1	-	15.54
GEN-VLKs [17][CVPR22]	R50	42.8	86.74	18.69
LOGICHOI (ours)	R50	49.8	89.65	16.84
Runtime Analysis. The computational complexity of our triplet-reasoning attention is squared compared to self-attention. Towards this, we make some specific designs: i) both the number of queries and Transformer decoder layers of our method are the smallest when compared to existing work [9-20], ii) as specified in §3.2, we filter the human, action, object queries and only keep half of them for efficiency, and iii) triplet-reasoning attention introduces few additional parameters. As summarized in Table 6, above facets make LOGICHOI even smaller in terms of Floating Point Operations (FLOPs) and faster in terms of inference compared to most existing work.
In this work, we propose LOGICHOI, the first Transformer-based neuro-logic reasoner for HOI detection. Unlike existing methods relying on predetermined human-object pairs, LOGICHOI enables the exploration of novel combinations of entities during decoding, improving effectiveness as well as the zero-shot generalization capabilities. This is achieved by i) modifying the self-attention mechanism in vanilla Transformer to reason over  $\langle$  human, action, object  $\rangle$  triplets, and ii) incorporating affordances and proxemics properties as logic-induced constraints to guide the learning and reasoning of LOGICHOI. Experimental results on two gold-standard HOI datasets demonstrate the superiority against existing methods. Our work opens a new avenue for HOI detection from the perspective of empowering Transformer with symbolic reasoning ability, and we wish it to pave the way for future research.
Acknowledgement. This work was partially supported by the Fundamental Research Funds for the Central Universities (No. 226-2022-00051) and CCF-Tencent Open Fund.
[1] Bangpeng Yao and Li Fei-Fei. Modeling mutual context of object and human pose in human-object interaction activities. In CVPR, 2010.
[2] Siyuan Qi, Wenguan Wang, Baoxiong Jia, Jianbing Shen, and Song-Chun Zhu. Learning human-object interactions by graph parsing neural networks. In ECCV, 2018.
[3] Ting-Chun Wang, Ming-Yu Liu, Jun-Yan Zhu, Andrew Tao, Jan Kautz, and Bryan Catanzaro. High-resolution image synthesis and semantic manipulation with conditional gans. In CVPR, 2018.
[4] Aishwarya Agrawal, Aniruddha Kembhavi, Dhruv Batra, and Devi Parikh. C-vqa: A compositional split of the visual question answering (vqa) v1. 0 dataset. arXiv preprint arXiv:1704.08243, 2017.
[5] Spencer Whitehead, Hui Wu, Heng Ji, Rogerio Feris, and Kate Saenko. Separating skills and concepts for novel visual question answering. In CVPR, 2021.
[6] Mitja Nikolaus, Mostafa Abdou, Matthew Lamm, Rahul Aralikatte, and Desmond Elliott. Compositional generalization in image captioning. In EMNLP, 2019.
[7] George Pantazopoulos, Alessandro Suglia, and Arash Eshghi. Combine to describe: Evaluating compositional generalization in image captioning. In ACL, 2022.
[8] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NeurIPS, 2017.
[9] Mingfei Chen, Yue Liao, Si Liu, Zhiyuan Chen, Fei Wang, and Chen Qian. Reformulating hoi detection as adaptive set prediction. In CVPR, 2021.
[10] Bumsoo Kim, Junhyun Lee, Jaewoo Kang, Eun-Sol Kim, and Hyunwoo J Kim. Hot: End-to-end human-object interaction detection with transformers. In CVPR, 2021.
[11] Masato Tamura, Hiroki Ohashi, and Tomoaki Yoshinaga. Qpic: Query-based pairwise human-object interaction detection with image-wide contextual information. In CVPR, 2021.
[12] Cheng Zou, Bohan Wang, Yue Hu, Junqi Liu, Qian Wu, Yu Zhao, Boxun Li, Chenguang Zhang, Chi Zhang, Yichen Wei, et al. End-to-end human object interaction detection with hoi transformer. In CVPR, 2021.
[13] Aixi Zhang, Yue Liao, Si Liu, Miao Lu, Yongliang Wang, Chen Gao, and Xiaobo Li. Mining the benefits of two-stage and one-stage hoi detection. In NeurlPS, 2021.
[14] Bumsoo Kim, Jonghwan Mun, Kyoung-Woon On, Minchul Shin, Junhyun Lee, and Eun-Sol Kim. Mstr: Multi-scale transformer for end-to-end human-object interaction detection. In CVPR, 2022.