Neural-Logic Human-Object Interaction Detection
Authors: Liulei Li
Jianan Wei
Wenguan Wang
Yi Yang
Venue: NeurIPS 2023 poster (2023)
Keywords: Human-Object Interaction
Neuro-Symbolic Computing
Compositional Generalization

Abstract: The interaction decoder utilized in prevalent Transformer-based HOI detectors typically accepts pre-composed human-object pairs as inputs. Though achieving remarkable performance, such a paradigm lacks feasibility and cannot explore novel combinations over entities during decoding. We present LogicHOI, a new HOI detector that leverages neural-logic reasoning and Transformer to infer feasible interactions between. entities. Specifically, we modify. self-attention mechanism in the vanilla Transformer, enabling it to reason over the ⟨ human, action, object ⟩ triplet and constitute novel interactions. Meanwhile, such a reasoning process is guided by two crucial properties for understanding HOI: affordances (the potential actions an object can facilitate) and proxemics (the spatial relations between humans and objects). We formulate these two properties in first-order logic and ground them into continuous space to constrain the learning process of our approach, leading to improved performance and zero-shot generalization capabilities. We evaluate L OGIC HOI on V-COCO and HICO-DET under both normal and zero-shot setups, achieving significant improvements over existing methods.

Neural-Logic Human-Object Interaction Detection
Liulei Li',
Jianan Wei",
Wenguan Wang?: Yi Yang²
'ReLER, AAlI, University of Technology Sydney *CCAL, Zhejiang University
https:://githb.. http://tww...o/eeeaaaanaaaaaaaaa ccm//veijiaannn//ooottoo))
Abstract
The interaction decoder utilized in prevalent Transformer-basee HOI detectors typically accepts pre-composed human-object pairs as inputs. Though achieving remarkable performance, such paradigm lacks feasibility and cannot explore novel combinations over entities during decoding. We present LOGICHOI, a new HOI detector that leverages neural-loge reasoning and Transformer to infer feasible interactions eeetweennntitienn Specificaally. wemodify the self-attention mechanism in vanilla Transformer, enabling to reason over the (human, action, object) triplet and constitute novel interactions. Meanwhile, such reasoning process is guided by erucial properties for understanding HOI: affordances (the potential actions an object can facilitate) and proxemics (the spatial relations between humans and objects). We formulate these two properties in first-order logic and ground them into continuous space to constrain the learning process of our approach, leading improved performance and zero-shot generalization capabilities. We evaluate LOGICHOI on V-COCO and HICO-DED under both normal and zero-shot setups, achieving significant improvements over existing methods.
Introduction
The main purpose of human-object interaction (HOI) detection is to interpret the intricate relationships between human and other objects within a given scene [1]. Rather than traditional visual perception tasks that focus on the recognition of objects or individual actions, HOI detection places a greater emphasis on reasoning over entities 22. and can thus benefit wide range of scene understanding tasks, including image synthesis
3]. visual question answering 4 []] and caption generation [6 7. etc.
Current top-leading HOI detection methods typically adopt Transformer (8] -baaed architecture, wherein the ultimate predictions are delivered by an interaction decoder. Such decoder takes consoli- dated embeddings of predetermined coman-objectt pairs [9 20 inputs, and it solely needs to infer the action or interaction categories of given entity pairs. Thoughachieving achieving remarkable performance over CNN-based work, such paradigm suffers from several limitations: first, the principal focus of them is to optimize the samples with known concepts, ignoring a large number of feasible combinations that never encountered within the training dataset, resulting in poor zero-shot generalization ability [2I]. Second, the human-object pairs are usually proposed by a simple MLP layer ( 22 37) or simultancously constructed when predicting humans and objects [10 I7]. without explicit modeling of the complex relationships among subjects, objects, and the ongoing interactions that happened between them. Third, existing methods lack the reasoning ability, caused by ignoring the key nature of HOI detection, i.e., the interaction should be mined between entities, but not pre-gives pairs.
In light of the above, propose to solve HOI detection via the integration of Transformer and logic-induced HOI relation learning, resulting in LOGICHOI, which enjoys the advantage of robust distributed representation of entities well principled symbolic reasoning [38 39]. Though
"Corresooning Author: Wenguan Wang.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

@a) affordances
Lannh
hh) proxemics
Left: self-attention aggregates information across pre- composed interaction queries. Middle: in contrast, our proposed triplet-reasoning attention traverses over human action ) and object ( queries propose plausible interactions. Right: logic-induced affordances and proxemics knowledge learning.
Figure I:
it was claimed that Seq2Seq models (e.g., Transformer) are inefficient to tackle visual reasoning problem [1)] models based on Transformer with improved architectures and training strategies have already been applied to tasks that require strong reasoning ability such as Sudoku (42), Raven's ' Progressive Matrices [43). compositional semantic parsing (44), spatial relation recognition [45). to name a few, and achieve remarkable improvements. Given the above success, we would like to argue that "Transformers are non-trivial symbolic reasoners, with only slight architecture changes and appropriate learning guidances", where the later imposes additional constraints for the learning and reasoning of modification. Specifically, for archilecture changes, we modify the attention mechanism in interaction decoder that sequentially associates each counterpart of the current query individually [
[ left), but encourage to operate in a triplet manner where states are updated by combining (human, action, object) three elements together. leading to triplet-reasoning attention (Fig. middle). This allows our model to directly reason over entity and the interaction they potentially constituted, therefore enhancing the ability to capture the complex interplay between humans and objects. To appropriately guide the learning process of such triplet reasoning in Transformer, we explore two HOI relations, affordances and proxemics. The former refers to that an object facilitates only a partial number of interactions, and objects allowing for executing a particular action is predetermined. For instance, the human observation of kite may give rise to imagined interactions such as launch or fly, while other actions such as repair, throw are not considered plausible within this context. In contrast, proxemics concerns the spatial relationships between humans and objects, e.g., when a human is positioned below something, the candidate actions are restricted to airplane, kite, etc. This two kind of properties are stated in first-order logical formulae and serve as optimization objectives of the outputs of Transformer. After multiple layers of reasoning, the results are expected to adhere to the aforementioned semantics and spatial knowledge, thereby compelling the model to explore and learn the reciprocal relations between objects and actions, eventually producing more robust and logical sound predictions. The integration of logic-guided knowledge learning serves as a valuable complement to triplet-reasoning attention, as it constrains triplet-reasoning atlention to focus on rule-satisfied triplets and discard unpractical combinations, enabling more effective and efficient learning, as well as faster convergence.
By enabling reasoning over entities in Transformer and explicitly accommodating the goal of pro- moting such ability through logic-induced learning, our method holds several appealing facets: first, accompanying reasoning-orient edge into Transformer in a logic-induced manner. In this way, our approach is simultaneously continuous neural computer and a discrete symbolic reasoner (albeit only implicitly), which meets the formulation of human cognition [7). Second, compared to neuro- syrroo-cmboiic methods solely driven by loss constraints [48 50). which are prone be over- over-fitting on supervised learning [42 and disregard reasoning at inference stage, we supplement it with task-specific architecture changes to facilitate more flexible inference (i.e., triplet-reasooinng attention). Additionally, the constraints are tailored to guide the learning of interaction decoders rather than the entire model, to prevent "cheating". " Third, our method does not rely on any discrete symbolic reasoners (e.g.. MLN (5T) or ProbLog 52 ) which increases the complexity of the model and cannot be jointly optimized with neural networks. Fourth, entities fed into Transformer are composed into interactions by the model automatically, such compositional manner contributes to improved zero-shot generalization ability.
architectural design, we embed affordances and proxemics knowl-
To the best of our knowledge, we are the first that leverages Transformer as the interaction reasoner. To comprehensively evaluate our method, we experiment it on two gold-standard HOI datasets (i.e., V-COCO and HICO-DET (44), where we achieve 35.47% % and 65.0% overall mAP score, setting new thee--o-- We also study the performance under the zero-shot setup from four different perspectives. As expected, our algorithm consistently delivers remarkable improvements, up to +5.16% % mAP under the unseen object setup, outperforming all competitors by a large margin.
2

---

2
Related Work
HOI Detection. Early CNN-based solutions for HOI detection can be broadly classified into two
paradigms: two-stage and one-stage. The two-stage methods (2 23 55 56) first detect entities by leveraging off-the-shelf detectors (ester among all possible human-object pairs. Though effective, they suffer from expensive computation due to the sequential inference architecture (17). and are highly dependent on prior detection results. In contrast, the one-stage methods (18-61) jointly detect human-object pairs and classify the interactions in an end-to-end manner by associating humans and objects with predefined anchors, which can be union boxes [58 61] interaction points [59 60). Despite featuring fast inference, they heavily rely to associate interactions with object detection results [10). Inspired by the advance of DETR 62). recent work [9-200 typically adopts Transformer-base architecture to predict interactions between humans and objects, which climinates complicated post-processing and generally demonstrates better speed-accuracy trade-off. To learn more effective HOI representations, several studies [17 19 63 66] also seek to transfer knowledge from visual-linguistic pre-trained
Faster R-CNN(57) and then determine the dense relationships
on hand-crafted
models (e.g., CLIP 67 which also enhances the zero-shot discovery capacity of models.
Although promising results have been achieved, they typically directly feed the pre-composed union representation of human-object pairs to the Transformer to get the final prediction, lacking reasoning over different entities to formulate novel combinations, therefore struggling with long-tailed distribu- tion and adapting unseen interactions. In our method, we handle HOI detection by triplet-reasoning within Transformer, where the inputs are entities. Such reasoning process is also guided by logic- induced affordances and proxemics properties learning with respect to the semantic and spatial rules, so as to discard unfeasible HOI combinations, and enhance zero-shot generalization abilities.
Computing (NSC) is a burgeoning research area teeks to seamlessly integrate the symbolic and statistical paradigms of artificial intelligence, while effectively inheriting the desirable characteristics of both (39]. Although the roots of NSC can be traced back to the seminal work of McCulloch and Pitts in 1943 (68). did not gain a systematic study until the 2000s, when a renewed interest in combining neural networks with symbolic reasoning emerged [51 69 73). These early methods often reveal limitations when attempting to handle large-scale and noisy data (74), as the effectiveness is impeded by highly hand-crafted rules and architectures. Recently, driven by both theoretical and practical perspectives, NSC has garnered increasing attention in holding the potential to model human cognition 38, 75. combine modern numerical connectionist with logic reasoning over abstract knowledge 48 81..1. so as to address the challenges of data efficiency [82] 83)
explainability 84887 87], and compositional generalization 88 89 in purely data-driven Al systems.
In this work, we address neuro-symbolic computing from two perspectives, first, we employ Trans- former as the symbolic reasoner but with slight architectural changes, ie., modifying the self-attention to triplet-reusoning attention and the inputs are changed from interaction pairs to human, action, and object entities, such relation interpretation task. Second, we condense affordances and proxemics properties into logical rules, which are further served to constrain the learning and reasoning of the aforementioned Transformer reasoner, making the optimization goal less ambiguous and mhowledge-ifformee
to empower the Transformer with strong reasoning ability for handling
Compositional Generalization, which pertains to the ability to understand and generate potentially boundless range of novel conceptual structures comprised of similar constituents (90). has long been thought to be the cornerstone of human intelligence (91]. For example, human can grasp the meaning of dax twice or dax and sing by learning the term dax (40), which allows for strong generalizations from limited data. In natural language processing, several efforts (22) have been made to endow neural networks with this kind of zero-shot generalization ability. Notably, the task proposed in [40]. referred to as SCAN, involves translating commands presented in simplified natural language (e.g., dax twice) into a sequence of navigation actions (e.g.. I I_DAX, I_DAX), Active investigations into visual compositional learning also undergo in the fields of image caption 66 7 [02) and visual question answering 4 103 104]. For instance, to effectively and explicitly ground entities, [102) first creates template with slots for images and then fills them with objects proposed by open-set detectors.
Though there has already been work 31 11.2 105 107] rethinking HOI detection from the perspec-
tive of compositional learning, they are, i) restricted to scenes with single object [105), J. ii) utilizing
compositionality for data augmentation (31 T06) or representation learning [32], without general-
ization during learning nor relation reasoning. A key difference in our work is that our method
3

---

arbitrarily constitute interactions as final predictions during decoding. This compositional learning and inference manner benefit the generalization to unseen interactions. Moreover, the affordances and proxemics properties can be combined, i.e., it is natural for us to infer that when human is above horse, the viable interactions options can only be sit_on and ride.
3 Methodology
In this section, we first review Transformer-bant HOI detection methods and the self-attention mechanism (3..)) Then, we elaborate the pipeline of our method and the proposed triplet-reasoning attention blocks ($3.2), which is guided by the logic-induced learning approach utilizing both affordances and proxemics properties ($.3) .. Finally, provide the implementation details ($3.4).
3.1 Preliminary: Transformer-based HOI Detection
Enlightened by the success of DETR 622 recent state-of-the-arts/ (9 20) for HOI detection typically adopt the -ncoder-decoder architecture based on Transformer. The key motivation shared across all of the above methods is that Transformer can effectively capture the long range dependencies and exploit the contextual relationships between human-object pairs 12] by means of self-attention. Specifically, interaction or action decoder is adopted, of which the query, key, value embeddings NDD are constructed from the unified embedding of human-objec pair Q* by:
F", Fk, F E R
F = ( (X+Q"1) ho W
Fh (X+ (X+Q" W F (X +
(1)
where X is the input matrix, WA, W W" E R are parameter matrices and Q Qn can be derived from the feature of union bounding box of human- commantoojectffe [2 or simply concatenating the embeddings of human and object together [17.. Then X is updated through self-attention layer by:
W Ex= softmax (F.FF")) -.
(2)
Here we adopt the single-head variant for simplification. Note that under this scheme, the attention is imposed over action or interaction embeddings, which has already been formulated before being fed into the action or interaction decoder. This raises two concerns i) may discard positive human- object pair and ii) cannot present novel combinations over entities during decoding.
3.2 HOI Detection via
Attention
In contrast to the above, we aim facilitate the attention over three key elements formulate an interaction (i.e., human, action on, object, therefore referring to triplet-reasoning attention), by leveraging the Transformer architecture. The feasible (human, action, object) tuples combined and filtered through the layer-wise inference within the Transformer. Towards this goal, we first adopt visual encoder which consists of a CNN backbone and Transformer encoder visual features V. Then, the learnable human queries Q* e R.N, xi action queries Qa e RR. XD and object queries Q' E T NoDD are fed into three parallel Transformer decoders DA, D DD to get the human, action, and object embeddings respectively by:
extract
Q = DVV Q"), Q DDVVVVVV Q"). ='= Q").
Here, No No and the superscripts are kept for improved clarity. All of the three query embeddings are then processed by linear layers to get the final predictions. For human and object queries, we supervise it with the class and bounding box annotations, while for the action queries, we only supervise them with image-level categories, i.e., what kinds actions are happened in this image. After that, we adopt interaction decoder DP composed by multiple Transformer layers in which the self-attention is replace by our proposed triplet-rease reasoning attention, so as to empower Transformer with the reasoning ability. Specifically, in contrast to 1q given Q", Q", Q", the input query, key, value embeddings F" F F" for triplet-reasoning attention are computed as:
F (X Q* Q*) W RR F* = (x+ (X+Q++Q)) W N.N.NN
(4)
F" W (X-Q*+0) ( E N.EBRNXX.NNNOO

---

qurrt
Poar
Took Chunan look glove)
X
abrect query
D
X
1.04
D
Overview of LOGICHOL. We first retrieve human, action, and object queries by D*, DD' and D", respectively. Then D' take them input, reasoning over entities and combining potential interaction triplets. Finally, such process is guided by affordances and proxemics properties, to be efficient and knowledge-informed.
Figure 2:
where @ is the element- wise production. Note that omit the dimension expanding operation for better visual presentation. Concretely, for F (X+ ++***++00 W the human queries Q" E No xD and action queries Q" E N.,, xD expanded to IR N₁ xlxD and R IxNNN xD respectively. In this manner, Q" + Q" associates each human and action entity, resulting in N₂, xNa human-action pairs in total. Na XN, viable action-object pairs risen by Q"+ Q in the same way. For the value embedding F", it encodes the representation of all N₂, xN x.N, potential interactions. Given specific element in " for instance, Finy. is composed from F and F.y. corresponding feasible interaction of embeddings
Q Q% and Qj. Then each element in inputs is updated by: X W" > softmax F /t///////// (..//DD--R FMij
(5)
where X' denotes the output of triplet-reasoning attention. In contrast self-atlention (cf Eq Eg our proposed priplet-reasonig attention (cf 55 stretches edges between every human-acti and action-obj ect pairs sharing the identical action query. By aggregating information from the relation between human-action and acti action-objed on-object, it learns to capture the feasibility of tuple (human, action, object) in compositional learning manner, simultaneously facilitating reasoning over entities. The final output Y of DD is given by:
Y = DV(V, Q*, Q", Q") N.N.NDD
(6)
which delivers the interaction prediction for N₁ xN, human-object pairs in total. We set No. No, N. relatively small number (e.g., ... 32) which is enough capture the entities in a single image, and larger number of queries will exacerbate the imbalance between positive and negative samples. Additionally, for efficiency, keep only half the number of human, object and action queries by filtering the low-scoring ones before sending them into the interaction decoder DD.
3.3 Logic-Guided Learning for Transformer-baser Reasoner
In this section, we aim to guide the learning and reasoning of LOGICHOI with the affordances and proxemics properties. Though there has already been several works concerning about these two prop- they typically analyze affordances from the statistical perspective, i.e., computing the distribution of co-occurrence of actions and objects so as to reformulate the predictions (33), simply integrate positional encodings into network features 15 22 generator (63] which introduces additional parameters. In contrast, we implement them from the perspective that the constrained subset is the logical consequence of pre-given objects or actions.
erties 15 33 63].
proposing two-path feature
Concretely, we first state these two kinds of properties as logical formulas, and then ground them into continuous space to instruct the learning and reasoning of our Transformer reasoner (i.e., interaction decoder D For proxemics, we define five relative positional relationships with human as the reference frame, which are above (e.g., kite above human), below (e.g., skateboard below human), around (e.g., giraffe around human), within (e.g., handbag within human), containing (e.g., bus containing human). To make the Transformer reasoner spatiality-awad the human and object embeddings retrieved from EqBaare concatenated with sinusoidal positional encodings generated from predicted bounding boxes. Then, given action and position relationship the set of infeasible interactions (i.e., triplet (human, action, object)) (h1, ... hM } can be derived:
(7) where x refer to one human-object pair that potential have interactions. In first-order logic, the semantics of variables (e.g., x) is usually referred to predicates (e.g., launch(r), above(r)). Eq
V=(((x)) A p(x) - -h (x) A -12(2r) ... -h (x));
5

---

states that, for instance, if the v is launch, and p is above, then in addition to interactions composed of actions apart from launch, human- maunch-boat should be included in {hr,... h,r as well. Similarly, given the object category o and position relationship we shall have:
Vz(o(z) A(zx) + (x) A -h2(2) A ... (x)
(8)
With 7 and Eq both affordances and proxemics properties, and the combination relationship between them are clearly stated. Next we investigate how to convert the above logical symbols into differentiable operation. Specifically, logical connectives (e.g., - 7, V, A) defined on discrete Boolean variables are grounded to functions on continuous variables using product logic (T08):
+ à = 1- v + t V = 2b + Φ 2 á
- v AΛΦ +b Φ
Similarly, the quantifier are implemented in generalized meen mean manner following [009]
=x(((x))) = ( Ex= (Tx))))
1
(10)
(x(v(x)) = =1 - E (1=
1
Given above relaxation, we ready to translate properties defined in first-order logical formulae into sub-symbolic numerical representations, so as to supervise the interactions {/1...., .ur predicted by the Transformer reasoner. For instance, Eq is grounded by Eq 9 and Eq 10
1 -up=1 = M EM ER (sx[u) sx
(11)
where 8& [u] and 8k [bm] refer to the scores of action e and interaction hm with respect to input sample Tk- Here K refers the number of all training samples and we relax it that in mini-batch. As mentioned above, the spatial locations of human and objects are concatenated into the query which means the spatial relation is predetermined and can be effortlessly inferred from the box predictions (details provided in Supplementary Materials). Thus, we omit p(x) in I1 Then, the action-posil tiitiiinnn loss is defined as: Le.p 1 Gu.p. p-
In a similar way, we can ground Eq 8 into:
Go.p =
1 Σk (sx (hml))))
(12)
where sk[o] refers to the score of object regarding to the input sample zy. The eeejje--posiiiie loss is defined as: Co.p = 1 go For Go.p. it scores the satisfaction of predictions to rules defined in 7 7 For example, given high probability of action ride (i.e., high value of s& (ul) and the position relationship is above, if the probability of infeasible interactions (e.g.. human-feed-ff fis") also high, then Go.p would receive low value as punish this prediction. Go.p is similar but scores the satisfaction of predictions to Eq with given position and objects such as horse, fish, etc. Through Eq. IT and [2 we aim achieve that, given the embeddings and locations of group of human and object entities, along with the potential actions happened within the image, the Transformer reasoner should speculate which pair of human-object engaged in what kind of interaction, while the prediction should respect to the rules defined in Eq and E
3.4 Implementation Details
Network Architecture. To make fair comparison with existing Transformer-based work 9 20) adopt ResNet-50 as the backbone. The visual encoder & is implemented as six Transformer encoder layers, while the three parallel human, object and action decoders Dd, D D are all constructed as three Transformer decoder layers. For the interaction decoder DP, we instantiate it with three Transformer decoder layers as well, but replacing self-aatention with our proposed triplet-reasoning attention. The number of human, object, action queries NA. No. Na is set to 32 for efficiency, and the hidden sizes of all the modules are set to 768. Since the state-of-the of-the-art work [7 19 63 66] usually leverages large-scale visual- linguiisi pre-trained models to further enhance the detection capability, we follow this setup and adopt CLIP 667.. To improve the inference efficiency of our framework, we further follow (17) which uses guided embeddings to decode humans and objects in a single Transformer decoder, Le., merging D and Do into unified one to simultaneously output both human and object predictions.

---

Training Objectives. LOGICHOI is jointly optimized by the HOI detection loss (i.e., CHOi) and logic-induced property learning loss (i.e., LLOG):
Lhoi + aLooG LLOG = + Lo.p.
(13)
Here a is set to 0.2 empirically. Note that LLOG solely update the parameters of the Transformer reasoner (i.e., interaction decoder DP) but not the entire network to prevent over-fitting. For LHoI. is composed of human/object (i.e., output of D' and D", respectively) detection loss, action (i.e.,
output of DD) classification loss well interaction (i.e., output of DD) classification loss.
4 Experiments
4.1 Experimental Setup
Datasets. We conduct experiments on two widely-used HOI detection benchmarks:
V-COCO [3 is carefully curated subset of MS-COCO [100] which contains 10,346 images (5,400 for training and 4,946 for testing). There are 263 human-object interactions annotated in this dataset in total, which are derived from 80 object categories and 29 action categories.
HICO-DET (54) consists of 47,776 images in total, with 38, 18 for training and 9,658 designated for testing. It has 80 object categories identical those in V-COCO and 117 action categories, consequently encompassing a comprehensive collection of 600 unique human-objec interactions.
Evaluation Metric. Following conventions 10 377, the mean Average Precision (mAP) is adopted for evaluation. Specifically, for V-COCO, includes all of the 29 action categories and scenario 2 (S2) which excludes 4 human body motions without interaction any objects. For HICO-DET, we perform evaluation across three category sets: all 600 HOI categories (Full), 138 HOI categories with less than 10 training instances (Rare), and the remaining 462 HOI categories (Non-Rare) Moreover, the mAP scores are calculated in two separate setups: i) the Default setup computing the mAP on all testing images, and ii) the Known Object setup measuring the AP for each object independently within the subset of images containing this object.
report the mAP scores under both scenario (S1) which
Zero-Shot HOI Detection. We follow the setup in previous work 17 21 31 106 107 to carry on zero- shot generalization experiments, resulting in four different settings: Rare First Unseen Combination (RF-UC), Non-rare First Unseen Combination (NF-UC), Unseen Verb (UV) and Unseen Object (UO) on HICO-DET. Specifically, the RF and NF strategies in the UC setting indicate selecting 120 most frequent and infrequent interaction categories for testing, respectively. In the UO setting, we choose 12 objects from 80 objects that are previously unseen the training set, while in the UV setting, we exclude 20 verbs from total of 117 verbs during training and only use them at testing.
Training. To ensure fair comparison with existing work 9 20
we initialize model with weights
of DETR 62) pre-trained on MS-COCO. Subsequently, we conducted training for 90 epochs using the Adam optimizer with a batch size of 16 and base learning rate le le-a, on 4 GeForce RTX 3090 GPUs. The learning rate is scheduled following a "step" policy, decayed by factor of 0.1 at the 60th epoch. line with [2 66). the random scaling aug- mentation is adopted, i.e., training images resized to maximum size of 1333 for the long edge, and min- imum size of 400 for the short edge.
Table 1: Comparison of zero-shot generalization with state-of-the-arts on
HICO-DET S4 test. See 442 for details.
Method
VL Pretrain
Type
Unseen
Full
VCL
31
RF-UC
10.06
24.28
21.43
ATL
107
RF-UC
9.18
24.67
21.57
FCL
T06
CVP9211
RE-UC
13.16
24.23
22.01
GEN-VLKT
T7
VV9922)
RF-UC
21.36
32.91
30.56
SCL
B2C
RF-UC
19.07
30.39
28.08
LOGICHOT
(ours)
RF-UC
25.97
34.93
33.17
VCL.
NF-UC
16.22
18.52
18.06
ATL
107
NF-UC
18.25
18.78
18.67
FCL
106
NF-UC
18.66
19.55
19.37
DEN-VLKT
NF-UC
25.05
23.38
23.71
SCL
NF-UC
21.73
25.00
24.34
LOGICHOT
(ours)
NF-UC
26.84
27.86
27.95
ATL 107
[VV19211
5.05
14.69
13.08
FCL
106
CVPR211
0.00
13.71
11.43
GEN-VLKI
CVP922)
10.51
28.92
25.63
LOGICHOl
(ours)
15.67
30.42
28.23
Testing. For fairness, we refrain from implementing
GEN-VLKT LOGICHOT (ours)
17
(CVP222
/
UV
20.96
30.23
28.74
UV
24.57
31.88
30.77

---

Table 2: Quantitative results HICO-DET 54
test and V-COCO 53)
test. See $4.3
for details.
Method
Backbone
VL Pretrain
Default Rare
Known Object
APsI
APS? role
Full
Non-Rare
Full
Rare
Non-Rare
iCAN
R50
14.84
10.45
16.150
16.26
11.33
17.73
45.3
UnionDef
R50
17.58
11.72
19.33
19.76
14.68
21.27
47.5
$6.2
PPDM
CVPR20)
HIG104
21.73
13.78
24.10
16.65
26.84
HOTR
10
ENPR211
R50
23.46
16.21
25.60
55.2
64.4
AS-Net
9
CVPR21)
R50
28.87
24.25
30.25
31.74
27.07
33.14
53,9
QPIC
[NPR21]
R50
29.07
21.85
31.23
31.68
24.14
33.93
58.8
61.0
CDN
R50
31.78
27.55
33.05
34.53
29.73
35,96
62.3
64.4
MSTR
R50
31.17
25.31
32.92
34.02
28.83
35.57
62.0
65.2
UPT
R50
31.66
25.94
33.36
35,05
29.27
36.77
59.0
64.5
STIP
CVPR221
32.22
28.15
33.43
31.43
IF-HOI [V9222
R50
33.51
30.30
34.46
36.28
33.16
37.21
63.0
65.2
ODM
B0CV22)
R50-FPN
31.65
24.95
33.65
Twin
TXCV22
R50-FPN
32.03
27.62
34.14
35.17
28.79
35.91
60.5
LOGICHOI (ours)
R50
34.53
31.12
35.38
37.04
34.31
37.86
63.7
64.9
CTAN
66
R50
31.71
24.82
33.77
33.96
26.37
36.23
60.1
SSRT
63
CVPR221
R50
30.36
25.42
31.83
63.7
65.9
DOQ
63
R50
33.28
29.19
34.50
63.5
GEN-VLK
CVPR22]
R50
33.75
29.25
35.10
37.80
34.76
38.71
62.4
64.4
LOGICHOI (ours)
R50
35.47
32.03
36.22
38.21
35.29
39.03
64.4
65.6
any data augmentation during testing. Specifically, first select K interactions with the highest scores and further filter them by applying NMS to retrieve the final predictions. Following the set K to 100.
convention 11 16 66).
4.2 Zero-Shot HOI Detection
The comparisons of our method against several top-leading zero-shot HOI detection models 17 106| [07] on HICO-DET test are presented in Table [ It can be seen that LOGICHOI outpertorms all of the competitors by clear margins across four different zero-shot setups.
Unseen Combination. As seen, LOGICHOI provides a considerable performance gain against existing methods. In partieular, outperforms the top-leading GEN-VLKT(17) by 4.61 % and 1.79% % in terms of mAP on unseen categories for RF and NF selections, respectively. These numerical results substantiate our motivation of empowering Transformer with the reasoning ability and guide the learning in a logic-induced compositional manner, rather than solely taking Transformer as an interaction classifier.
Unseen Object. Our approach achieves dominant results under the UO setting, surpassing other competitors across all metrics. Notably, it yields unseen mAP 15.67 % and overall mAP 28.23%. while the corresponding scores for the SOTA methods [17] are 5.05%, 13.08% and 10.51%, 25.63%, presenting an improvement up 5.16% mAP for unseen categories. This reinforces belief that empowering Transformer with logic- guided reasoning ability is imperative and indispensable.
Unseen Verb. Our method also demonstrates superior performance in the UV setup. Concretely, it surpasses GEN-VLKT 17 by 3.61% mAP on unseen categories and achieves 30.77 overall scores.
All of the above improvement on zero-shot generalization confirms the effectiveness of our proposed Transformer reasoner which learns compositional manner, and is informed by affordances and proxemics knowledge to address novel challenges that was never encountered before.
4.3 Regular HOI Detection
HICO-DET. In Table we present the results of LOGICHOI and other top-performing models under the normal HOI detection setup. Notably, on HICO-DET (54) test, our solution demonstrates significant improvement over previous state-of-the-art [17). with substantial margin of /
2.78% 11.1%% mAP for Full, Rare, and Non-Rare categories, under the Default setup. Moreover, in
terms of Known Object, method attains exceptional mAP scores of 38.21 % /3 /3/9003%%

---

Table 3: Analysis of essential components of LOGI- cHOI on HICO-DET (54). See $4.4 for details.
Table 4: Analysis of LRI. under the zero-shot setup
of unseen categories. See $4.4 for details.
TRA
LRL
Full
Rare
Non-Rare
Setting
RF-UC
UO
UV
31.87
26.14
33.29
TRA 24.01
13.26
23.14
34.32
30.67
35.19
+ +hopp
25.22
15.32
23.68
33.26
29.53
34.56
+houp
25.34
13.91
24.29
/
35,47-3.00
12.03
36.2
LOGICHOI
25.97
15.67
24.57
Table 5: Analysis of number of decoder layer and query on HICO-DET 54)
test. See $4.4
for details.
# of layers (L)
Full
Rare
Non-Rare
# of queries (N)
Full
Rare
Non-Rare
2
34.61
30.72
35.54
16
35.06
31.36
35.94
3
35.47
32.03
36.22
32
35.47
32.03
36.22
4
35.37
31.96
36.09
64
35,26
31.65
36.06
6
35.61
32.13
36.39
128
34.67
30.98
35.53
(a) number of interaction decoder layer.
(h) number of the queries.
As indicated by the last two columns of Table competitive models on V-COCO [33] test. Despite the relatively smaller number of images and HOI categories this dataset, our method still yields promising results, showcasing its effectiveness. In particular, it achieves mean mAP score of 65.0 % across two scenarios.
V-COCO.
also compare LOGICHOI with
4.4 Diagnostic Experiment
For in-depth analysis, we perform a series of ablative studies on HICO-DET (4] test.
Key Component Analysis. We first examine the efficiency of essential designs of LOGICHOI, i.e., triplet-ressoning attention (TRA) and logic-guided reasoner learning (LRL), which is summarized in Table Three crucial conclusions can be drawn. First, our proposed triplet-reusoningg atlention leads to sigmificant performance improvements against the baseline across all the metrics. Notably, TRA achieves 4.53 % mAP improvement Rare congeries, demonstrating the ability of our Transformer Reasoner to reason over entities and generate more feasible predictions. Second, we also observe compelling gains from incorporating logic-guided reasoner learning into the baseline, even with basic self-attention, affirming its versatility. Third, our full model LOGICHO) achieves the satisfactory performance, confirming the complementarity and effectiveness of our designs.
Logic-Guided Learning. We guide the learning of Transformer reasoner with two logic-induced properties. Table Table/rreport the related scores of unseen categories under three zero-shot setups. The contributions of L..pp and Lo.p are approximately equal the RF-UC setup, since during training, all actions and objects be seen and utilized guide reasoning. On the other hand, under the UO and UV setups, the improvements heavily rely on Lr,p and Co.p respectively, while the other one brings minor enhancements. Finally, the combination of them leads to LOGICHOI, the new state-of-the the-thrrttt...
Number of Decoder Layer. We fur- ther examine the effect of the num- ber Transformer decoder layer used in DP, As shown in Table 5a LOGI- CHOI achieves similar performance when L. is larger than 2. For efficiency, we set L = which is the smallest among existing work (0].
Table 6: Comparison of parameters and rumning efficiency.
Method
Backbone
Params
FLOPs
FPS
Two-stages
Detectors:
iCAN [II
[MVC11
R50
39.8
5.99
DRG
BCCV20]
R50-FPN
46.1
6.05
SCG
ICCV211
R50-FPN
53.9
7.13
STIP
[eV9R22)
R50
50.4
6.78
One-stages
Detectors:
Number of Query. Next we probe the impact of the number of query for human, object and action in Yable(5b) Note that the number of these three kind of queries identical and we re- fer it to N. The best performance is obtained at N 32 and more queries lead to inferior performance.
PPDM
59
[CVeR20)
HG104
194.9
17.14
HOTR
10
R50
51.2
90.78
15.18
HOlTrans
R50
41.4
87.69
18.29
AS-Net
CVPR211
R50
52.5
87.86
QPIC
[VPR21)
R50
41.9
88.87
16.79
CDN-S
NartiS211
R50
42.1
15.54
GEN-VLK
[CVRR22)
R50
42.8
86.74
18.69
LOGICHOI (ours)
R50
49.8
89.65
16.84

---

Runtime Analysis. The computational complexity of our triplet-reasoning attention is squared compared self-attention. Towards this, we make some specific designs: i) both the number of queries and Transformer decoder layers of our method are the smallest when compared to existing work (9)20), ii) as specified in $3.2 we filter the human, action, object queries and only keep half of them for efficiency. and iii) triplet-reasoning attention introduces few additional parameters. As summarized in Tablel6, above facets make LOGICHOI even smaller in terms of Floating Point Operations (FLOPs) and faster in terms of inference compared to most existing work.
5 Conclusion
In this work, propose LOGICHOI, the first Transformer-basee neuro-logic reasoner for HOI detection. Unlike existing methods relying on predetermined human-object pairs, LOGICHOI enables the exploration of novel combinations of entities during decoding, improving effectiveness as well as the zero-shot generalization capabilities. This is achieved by i) modifying the self- atlention mechanism in vanilla Transformer reason over (human, action, object) triplets, and ii) incorporating affordances and proxemics properties as logic-induced constraints to guide the learning and reasoning of LOGICHOI. Experimental results on two gold-standard HOI datasets demonstrates the superiority against existing methods. Our work opens a new avenue for HOI detection from the perspective of empowering Transformer with symbolic reasoning ability, and we wish it pave the way for future research.
Acknowledgement. This work was partially supported by the Fundamental Research Funds for the Central Universities (No. 226-2022-00051) and TCF-eennent Open Fund.
References
[1] Bangpeng Yao and Li Fei-Fei. Modeling mutual context of object and human pose in human-object interaction activities. In CVPR, 2010.
[2] Siyuan Qi, Wenguan Wang, Baoxiong Jia, Jianbing Shen, and Song-Chun Zhu. Learning human-ohject interactions by graph parsing neural networks. In ECCV, 2018.
[3] Ting-Chun Wang, Ming-Yi Yu Liu, Jun-Yan Zhu, Andrew Tao, Jan Kautz, and Bryan Catanzaro. High- resolution image synthesis and semantic manipulation with conditional gans. In CVPR, 2018.
[4] Aishwarya Agrawal, Aniruddha Kembhavi, Dhruv Batra, and Devi Parikh. C-vqa: A compositional split of the visual question answering (vqa) v1. dataset. arXiv preprint 17Xx::770..0843 2017.
[5] Spencer Whitehead, Hui Wu, Heng Ji. Rogerio Feris, and Kate Saenko. Separating skills and concepts for novel visual question answering. In CVPR, 2021.
[6] Mitja Nikolaus, Mostafa Abdou, Matthew Lamm, Rahul Aralikatte, and Desmond Elliott. Compositional generalization in image captioning. In EMNLP, 2019.
[7] George Pantazopoulos. Alessandro Suglia. and Arash Eshghi. Combine to describe: Evaluating composi- tional generalization in image captioning. In ACL, 2022.
[8] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NeurlPS, 2017.
[9] Mingfei Chen, Yue Liao, Si Liu, Zhiyuan Chen, Fei Wang, and Chen Qian. Reformulating hoi detection adaptive set prediction. In CVPR, 2021.
[10] Bumsoo Kim, Junhyun Lee, Jaewoo Kang, Eun-Sol Kim, and Hyunwoo J Kim. Hotr: End-to-end human-object interaction detection with transformers. In CVPR, 2021.
[11] Masato Tamura, Hiroki Ohashi, and Tomoaki Yoshinaga. Qpic: Query-hased pairwise human-object interaction detection with wmdee-wide contextual information. In CVPR, 2021.
[12] Cheng Zou, Bohan Wang, Yue Hu, Junqi Liu, Qian Wu, Yu Zhao, Boxun Li, Chenguang Zhang, Chi Zhang, Yichen Wei, et al. End-to-end human object interaction detection with hoi transformer. In CVPR, 2021.
[13] Aixi Zhang, Yue Liao, Si Liu, Miao Lu, Yongliang Wang, Chen Gao, and Xiaobo Li. Mining the benefits of two-stage and stage hoi detection. In NeurIPS, 2021.
[14] Bumsoo Kim, Jonghwan Mun, Kyoung- Woon On, Minchul Shin, Junhyun Lee, and Eun-Sol Kim. Mstr: Multi-scale transformer for end-to-end hujen-object interaction detection. In CVPR, 2022.
Kyonng-Woon
10

---

[15] Frederic Z Zhang, Dylan Campbell, and Stephen Gould. Efficient two-stage detection of humen- interactions with novel unary- transformer. In CVPR, 2022.
[16] Desen Zhou, Zhichao Liu, Jian Wang, Leshan Wang, Tao Hu, Errui Ding, and Jingdong Wang. Human- object interaction detection via disentangled transformer. In CVPR, 2022.
[17] Yue Liao, Aixi Zhang, Miao Lu, Yongliang Wang. Xiaoho Li, and Liu. Gen-vlkt: Simplify association
and enhance interaction understanding for hoi detection. In CVPR, 2022.
[18] Xinpeng Liu, Yong-Lu Li, Xiaoqian Wu, Yu-Wing fu-Wing Tai, Cewu Lu, and Chi-Kcung Tang. Interactiveness field in human-object interactions. In CVPR, 2022.
[19] Hangjie Yuan, Jianwen Jiang, Samuel Albanie, Tao Feng. Ziyuan Huang. Dong Ni, and Minggian Tang. Rlip: Relational language-imin -magge pre- uve-trainin for human object interaction detection. In NeurlPS, 2022.
[20] Xubin Zhong, Changxing Ding, Zijian Li, and Shaoli Huang. Towards hard-positive query mining for detr-based human-object interaction detection. In ECCV, 2022.
[21] Zhi Hou, Baosheng Yu, and Dacheng Tao. Discovering human-object interaction concepts via self- compositional learning. In ECCV, 2022.
[22] Yong Zhang. Yingwei Pan, Ting Yao, Rui Huang. Tao Mei, and Chang- Wen Chen. Exploring structure- aware transformer over interaction proposals for buman-object interaction detection. In CVPR, 2022.
[23] Tanmay Gupta, Alexander Schwing, and Derek Hoiem. No-frills object interaction detection:
Factorization, layout encodings, and training techniques. In JCCV, 2019.
[24] Bo Wan, Desen Zhou, Yongfei Liu, Rongjie Li, and Xuming He. Pose-aware multi-level feature network for human object interaction detection. ICCV, 2019.
[25] Bingjie Xu, Yongkang Wong, Junnan Li, Qi Zhao, and Mohan S Kankanhalli. Learning to detect human-object interactions with knowledge. In CVPR, 2019.
[26] Penghao Zhou and Mingmin Chi. Relation parsing neural network for human-object interaction detection. In ICCV, 2019.
[27] Yong-Lu Lu Li, Siyuan Zhou, Xijie Huang, Liang Xu, Ze Ma, Hao-Shu Fang. Yanfeng Wang, and Cewu Lu. Transfcrable interactiveness knowledge for human-object interaction detection. In CVPR, 2019.
[28] Oytun Ulutan, ASM Iftekhar, and Bangalore S Manjunath. Vsgnet: Spatial attention network for detecting human object interactions using graph convolutions. In CVPR, 2020.
[29] Hai Wang, Wei-shi Zheng, and Ling Yinghiao. Contextual heterogeneous graph network for human-object interaction detection. In ECCV, 2020.
[30] Chen Gao, Jiarui Xu, Yuliang Zou, and Jia-Bin Huang. Drg: Dual relation graph for human-object interaction detection. In ECCV, 2020.
[31] Zhi Hou, Xiaojiang Peng, Yu Qiao, and Dacheng Tao. Visual compositional learning for human-object interaction detection. In ECCV, 2020.
[32] Yong-Lu Li, Xinpeng Liu, Xiaoqian Wu, Yizhuo Li, and Cewu Lu. Hoi analysis: Integrating and decomposing human-object interaction. In NewrlPS, 2020.
[33] Dong-Jin Kim, Xiao Sun, Jinsoo Choi, Stephen Lin, and In So Kweon. Detecting human-object interactions with action co-occurrence priors. In ECCV, 2020.
[34] Yong-Lu Li, Xinpeng Liu, Han Lu, Shiyi Wang, Junqi Liu, Jiefeng Li, and Cewu Lu. Detailed 2d-3d joint representation for human-object interaction. CVPR, 2020.
[35] Yang Liu, Qingchao Chen, and Andrew Zisserman. Amplifying key cues for human-chhjcctinnterattio detection. In ECCV, 2020.
[36] Frederic Z Zhang, Dylan Campbell, and Stephen Gould. Spatially conditioned graphs for detecting human-object interactions. In ICCV, 2021.
[37] Georgia Gkioxari, Ross Girshick, Piotr Dollár, and Kaiming He. Detecting and recognizing human-object interactions. In CVPR, 2018
[38] Yoshihiro Maruyama. Symbolic and statistical theories of cognition: towards integrated artificial intelli- gence. In International Conference on Software Engineering and Formal Methods, 2020.
[39] Luc De Raedt, Sehastijan Dumančić, Robin Manhaeve, and Giuseppe Marra. From statistical relational
to neural-symbolic artificial intelligence. In IJCAI, 2021.
[40] Brenden Lake and Marco Baroni. Generalization without systematicity; On the compositional skills of sequence-to-seqqencce recurrent networks. In ICML,, 2018
[41] Jasmijn Bastings, Marco Baroni, Jason Weston, Kyunghyun Cho, and Douwe Kiela. Jump to better conclusions: Scan both left and right. In EMNLP, 2018.

---

[42] Cristina Cornelio, Jan Stuehmer. Shell Xu Hu, and Timothy Hospedales. Learning where and when to reason in neuro-symbolic inference. In ICLR, 2023.
[43] Shanka Subhra Mondal, Taylor Whittington Webb, and Jonathan Cohen. Learning to reason over visual objects. In ICLR, 2023.
[44] Santiago Outanon. Joshua Ainslie, Zachary Fisher, and Vaclav Cvicek. Making transformers solve compositional tasks. In ACL, 2022.
[45] Kenneth Marino, Xinlei Chen, Devi Parikh, Abhinav Gupta, and Marcus Rohrbach. Krisp: Integrating implicit and symbolie knowledge for open-domain knowledge- vqa. In CVPR, 2021.
[46] Paul Smolensky. On the proper treatment of connectionism. Behavioral and brain sciences, 11(1):1 1988.
[47] Paul Smolensky, Richard McCoy, Roland Femanddzz, Matthew Goldrick, and Jianfeng Gao. Neurocompo- sitional computing: From the central paradox of cogmition to a new generation of ai systems. A/ Magazine, 43(3):308-322
2022.
[48] Zhiting Hu, Xuezhe Ma, Zhengzhong Liu, Eduard Hovy, and Eric Xing. Harnessing deep neural networks with logic rules. In ACL, 2016.
[49] Jingyi Xu, Zilu Zhang, Tal Friedman, Yitao Liang, and Guy Breeck. A semantic loss function for deep learning with symbolic knowledge. ICML., 2018.
[50] Yaqi Xie, Ziwei Xu, Mohan S Kankanhalli, Kuldeep S Meel, and Harold Soh. Embedding symbolic knowledge into deep networks. In NeurlPS, 2019.
[51] Matthew Richardson and Pedro Domingos. Markov logic networks. Machine learning, 62:107-136, 2006.
[52] Luc De Racdt, Angelika Kimmig, and Hannu Toivonen, Problog: A prohabilistic prolog and its application in link discovery. In ICAI,, 2007.
[53] Saurabh Gupta and Jitendra Malik. Visual semantic role labeling. arXiv preprint 1rXiv:1505.04474. 2015.
[54] Yu-We Wei Chao, Yunfan Liu, Xieyang Liu, Huayi Zeng, and Jia Deng. Learning to detect human-object interactions. In WACV, 2018.
[55] Tianfei Zhou, Wenguan Wang. Siyuan Qi, Haibin Ling, and Jianbing Shen. Cascaded human-object interaction recognition. In CVPR, 2020.
[56] Tianfci Zhou, Siyuan Qi. Wenguan Wang, Jianbing Shen, and Song-Chun Zhu. Cascaded parsing of human-object interaction recognition. IEEE TPAMI, 44(6):2827-2888 2021.
[57] Shaoqing Ren, Kaiming He, Ross Girshick. and Jian Sun. Faster r-cnn: Towards real-time object detection with region proposal networks. NeurlPS, 2015.
[58] Bumsoo Kim, Tacho Choi, Jacwoo Kang, and Hyunwoo J Kim. Uniondet: Union-level detector towards real-time human-object interaction detection. In ECCV, 2020.
[59] Yue Liao, Si Liu, Fei Wang. Yanjie Chen, Chen Qian, and Jiashi Feng. Ppdm: Parallel point detection and matching for real-time human-object interaction detection. In CVPR, 2020.
[60] Tiancai Wang, Tong Yang, Martin Danelljan, Fahad Shahbaz Khan, Xiangyu Zhang, and Jian Sun.
I Learning human-object interaction detection using interaction points. In CVPR, 2020.
[61] Hao-Shu Fang, Yichen Xie, Dian Shao, and Cewu Lu. Dirv: Dense interaction region voting for end-to-end human-object interaction detection. In AAAI, 2021.
[62] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In ECCV, 2020.
[63] ASM Iftckhar, Hao Chen, Kaustav Kundu, Xinyu Li, Joseph Tighe, and Davide Modolo. What to look at and where: Semantic and spatial refined transformer for detecting human-object interactions. In CVPR, 2022.
[64] Suchen Wang, Yueqi Duan, Henghui Ding, Yap-Peng Tan, Kim-Hui Yap, and Junsong Yuan. Learning transferable human- -ojjcct interaction detector with natural language supervision. In CVPR, 2022.
[65] Xian Qu. Changxing Ding, Xingao Li, Xubin Zhong, and Dacheng Tao. Distillation using oracle queries for transformer-bant human-object interaction detection. CVPR, 2022.
[66] Leizhen Dong, Zhimin Li, Kunlun Xu, Zhijun Zhang, Luxin Yan. Sheng Zhong, and Xu Zou. Category- aware transformer network for better human-object interaction detection. In CVPR, 2022.
[67] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin. Jack Clark, et al. Learning transferable visual models from natural language supervision. In ICML, 2021.
12

---

[68] Warren S McCulloch and Walter Pitts. A logical calculus of the ideas immanent in nervous activity. The Bulletin Mathematical Biophysics, 5:113-133, 1943.
[69] Geofrey G Towell, Jude W Shavlik, Michiel O Noordewier, et al. Refinement of approximate domain theories by knowledge-hased neural networks. In Proceedings of the National Conference on Artificial Intelligence, 1990.
[70] Jordan B Pollack. Recursive distributed representations. Artificial Intelligence, 46( 16))777777 1990.
[71]
Lokendra Shastri and Venkat Ajjanagadde. From simple associations systematic reasoning: A connec- tionist representation of rules, variables and dynamic bindings using temporal synchrony, Behavioral and Brain Sciences, 16(3):417-451, 1993.
[72] Avila Garcez and Gerson Zaverucha. The connectionist inductive learning and logic programming system. Applied Intelligence Journal, 11(1):59-7) 77, 1999.
[73] Antony Browne and Ron Sun. Connectionist inference models. Neural Networks, 14(10):131--3555
2001.
[74] Shaoyun Shi, Hanxiong Chen, Weizhi Ma, Jiaxin Mao, Min Zhang, and Yongfeng Zhang. Neural logic
reasoning. In International Conference on Information & Knowledge Management, 2020.
[75] Wenguan Wang, Yi Yang, and Fei Wu. Towards data-and and knowledge-drine survey on neuro-symholic computing. arXiv preprint arXiv:22 10. 2022.
artificial intelligence: A
[76] Russell Stewart and Stefano Ermon. Label-free supervision of neural networks with physics and domain knowledge. In AAAI, 2017.
[77] Tim Rocktäschel and Sebastian Riedel. End-to-end differentiable proving, NewrlPS, 2017.
[78] Mare Fischer, Mislav Balunovic, Dana Drachsler-Cohen, Timon Gehr, Ce Zhang, and Martin Vechev. DI2: Training and querying neural networks with logic. In ICML, 2019.
[79] Liulei Ii, Wenguan Wang, and Yi Yang. Logicseg: Parsing visual semantics with neural logic learning and reasoning. In ICCV, 2023.
[80] Chen Liang, Wenguan Wang, Jiaxu Miao, and Yi Yang. Logic-induced diagnostic reasoning for semi- supervised semantic segmentation. In /CCV, 2023.
[81] Lifeng Fan, Wenguan Wang, Siyuan Huang, Xinyu Tang, and Song-Chun Zhu. Understanding human gaze communication by spatio-temporal graph reasoning. In ICCV, 2019.
[82] Daoming Lyu, Fangkai Yang, Bo Liu, and Steven Gustafson. Sdrl: interpretable and data-efficient deep reinforcement learning leveraging symbolic planning. In AAAI, 2019.
[83] Kexin Yi, Jiajun Wu, Chuang Gan, Antonio Torralba, Pushmeet Kohli, and Josh Tenenbaum. Neural- symbolic vqa: Disentangling reasoning from vision and language understanding. In NeurlPS, 2018.
[84] Yuan Yang and Le Song. Leam to explain efficiently via neural logic inductive learning. In ICLR, 2019.
[85] Richard Evans and Edward Grefenstette. Learning explanatory rules from noisy data. Journal of Artificial Intelligence Research, 61:1-66 64, 2018.
[86] Jiaxin Shi, Hanwang Zhang, and Juanzi Li. Explainable and explicit visual reasoning over scene graphs. CVPR, 2019.
[87] Liulei Li, Tianfei Zhou, Wenguan Wang, Jianwu Li, and Yi Yang. Deep hierarchical semantic segmenta- tion. In CVPR, 2022.
[88] Maxwell Nye, Armando Lezama, Josh Tenenbaum, and Brenden M Lake. Learning compositional rules via neural program synthesis. In NeurlPS, 2020.
[89] Xinyun Chen, Chen Liang. Adams Wei Yu. Dawn Song, and Denny Zhou. Compositional generalization via neural-symbolic stack machines. In NeurlPS, 2020
[90] Daniel Keysers, Nathamael Schärli, Nathan Scales. Hylke Buisman, Daniel Furrer, Sergii Kashubin. Nikola Momehev, Danila Sinopalnikov, Lukasz Stafiniak, Tibor Tihon, et al. Mcasuring compositional generalization: A comprehensive method on realistic data. In ICLR, 2020.
[91] Jerry A Fodor and Zenon Pylyshyn. Connectionism and cognitive architecture: A critical analysis.
Cognition, 28(1- 22:3-71,1 1988.
[92] Najoung Kim and Tal Linzen. Cogs: A compositional generalization challenge based on semantic interpretation. In EMNLP. 2020.
[93] Brenden M Lake. Compositional generalization through meta coaenncc-tt-squeene learning. In NewrlPS, 2019.
[94] Daniel Furrer, Mare van Zee, Nathan Scales, and Nathanael Schärli. Compositional generalization in semantic parsing: Pre-training vs. specialized architectures. arXiv preprint 2rX0v:2007.08970, 2020.
13

---

[95] Jonathan Herzig and Jonathan Berant. Span-based semantic parsing for compositional generalization. In ACL, 2021.
[96]
Heary Conklin, Bailin Wang, Kenny Smith, and Ivan Titov. Meta-learning to compositionally generalize. In ACL, 2021.
[97]
Ekin Akyürek, Afra Feyza Akyürek, and Jacob Andreas. Learning to recombine and resample data for compositional generalization. In ICLR, 2021.
[98] João Loula, Marco Baroni, and Brenden Lake. Rearranging the familiar: Testing compositional general- ization in recurrent networks. EMNLP, 2018.
[99] Inbar Oren, Jonathan Herzig, Nitish Gupta, Matt Gardner, and Jonathan Berant. Improving compositional generalization in semantic parsing. In EMNLP, 2020.
[100] Peter Shaw, Ming-Wai Chang, Panupong Pasupat, and Kristina Toutanova. Compositional generalization and natural language variation: Can a semantic parsing approach handle both? In ACL, 2021.
[101] Yuval Atzmon, Jonathan Berant, Vahid Kezami, Amir Globerson, and Gal Chechik. Learning to generalize new compositions in image understanding. arXiv preprint arXiv: 108. 07639, 2016.
[102] Jiasen Lu, Jianwei Yang, Dhruv Batra, and Devi Parikh. Neural haby talk. In CVPR, 2018.
[103] Dzmitry Bahdanau, Shikhar Murty, Michael Noukhovitch, Thien Hum Nguyen, Harm de Vries, and Aaron Courville. Systematic generalization: What required and can it be learned? In ICLR, 2019.
[104] Justin Johnson, Bharath Hariharan, Laurens Van Der Maaten, Li Fei-Fei, C Lawrence Zitnick, and Ross Girshick. Clevr: diagnostic dataset for compositional language and elementary visual reasoning. In CVPR, 2017.
[105] Keizo Kato, Yin Li, and Abhinav Gupta. Compositional learning for human object interaction. In ECCV, 2018.
[106] Zhi Hou, Baosheng Yu, Yu Qiao, Xiaojiang Peng, and Dacheng Tao. Detecting human-object interaction via fabricated compositional learning. In CVPR, 2021.
[107] Zhi Hou, Baosheng Yu, Yu Qiao, Xiaojiang Peng. and Dacheng Tao. Affordance transfer learning for human-obie object interaction detection. In CVPR, 2021.
[108] Emile van Krieken, Erman Acar, and Frank van Harmelen. Analyzing differentiable fuzzy logic operators. Artificial Intelligence, 302:10366: 2022.
[109] Samy Badreddine, Artur d' Avila Garcez, Luciano Scrafini, and Michael Spranger. Logic tensor networks. Artificial Intelligence, 303:300::000 103649, 2022.
[110] Tsung-Yi Yi Lin, Michael Maire, Serge Belongie, James Hays, Pictro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Mierosoft coco: Common objects in context. In ECCV, 2014.
[111] Chen Gao, Yuliang Zou, and Jia-Bin Huang. ican: Instance-centric attention network for human-object interaction detection. In BMVC, 2018.
[112] Guangzhi Wang. Yangyang Guo, Yongkang Wong, and Mohan Kankanhalli. Chairs be stood on: Overcoming object bias in human-object interaction detection. In ECCV, 2022.
[113] Danyang Tu, Xiongkuo Min, Huiyu Duan, Guodong Guo, Guangtao Zhai, and Wei Shen. Twin; Human- object interaction detection via transformer with irregular windows. In ECCV, 2022.
14


Reviews:
[{"review_id": "Kr9PK6feiV", "reviewer": "Reviewer_mDiH", "summary": "The paper presents LoTR, a method for detecting human-object interactions (HOI) that utilizes Transformer as a reasoning mechanism to model the interactions between people, objects and their interactions. By feeding the self-attention with different feature combination as key query and value, LoTR can infer triplets and establishes meaningful interactions. The paper demonstrates the effectiveness of LoTR through experimental results on two widely used datasets, showcasing its strong performance.", "questions": "Please see my comments above.", "limitations": "None", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper is well written and well motivated.\n- The idea of Triplet-Reasoning Attention is well motivated and seems working as expected.\n- The results are comprehensive (I do have concerns on the results as well, please see section below).", "weaknesses": "1. Some of the claims may lack sufficient support or appear too strong. For instance, in lines 30-33, it is advisable to provide citations or evidence to substantiate the claim. It seems contradictory to assert that the model can accurately identify the correct pair while lacking comprehension of them.\n\n2. The comparison of results in Table 1 and Table 2 lacks fairness. As indicated in line 249, this paper utilizes the CLIP model, whereas most of the baselines are trained on significantly smaller datasets. The authors should establish a fair comparison by employing the exact same backbone or explicitly state the pre-training set used for the backbones.\n\n3. Additional discussion and analysis are necessary. For instance, when the proposed model performs worse than STIP on V-COCO, what could be the potential reasons? Currently, the paper criticizes STIP for its performance on other datasets but fails to explain why the proposed method does not generalize as well to V-COCO.\n\n4. In experimental results section, more details are required. For example, what does the bolded text signify in Table 5? Additionally, when the authors claim that they selected a three-layers decoder for improved efficiency, what are the corresponding FLOPs or latencies associated with different configurations?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1: Claims may be too strong and lack sufficient support.**\n\n**A1:** Thank you for pointing this out! The related parts will be revised as follows: \n\n\"**First**, the principal focus of them is to optimize the samples with known concepts, ignoring a large number of feasible combinations that were never encountered within the training dataset, resulting in poor zero-shot generalization ability [ref1]. **second**, the human-object pairs are usually proposed by a simple MLP layer[21,39-53] or simultaneously constructed when predicting humans and objects[10-17], without explicit modeling of the complex relationships among subjects, objects, and the ongoing interactions that happened between them.\"\n\nFor the first limitation, we supplement the training objectives with logic-guide constraints. These constraints are converted from the affordances and proxemics properties, effectively furnishing the model with ongoing guidance, even for instances never seen before. \n\nFor the second limitation, we decouple the inputs of interaction decoders into three different kinds of entities (*i.e.*, \"human\", \"object\" and \"action\"), and the final predictions (*i.e.*, \"human-action-object\") are then obtained through the application of triplet-reasoning attention, which takes these three items as input and facilitates reasoning over them.\n\n[ref1] Discovering Human-Object Interaction Concepts via Self-Compositional Learning. ECCV 2022.\n\n---\n\n**Q2: Comparison to work without CLIP.**\n\n**A2:** Good suggestion! To enable a fair comparison, we additionally experiment with our proposed method without using visual-linguistic pre-trained models like CLIP. The results under the zero-shot setup are summarized below. It emerges that even in the absence of CLIP, LoTR can still produce top-leading performance. Please note that FCL[98], ATL[99], and SCL[100] are **meticulously tailored** for zero-shot HOI detection, entailing much more intricate training strategies such as fabricated pair generation, pseudo labeling, and self-supervised learning, *etc.*\n\n|Method|Type|VL Pretrain|Unseen|Seen|Full|\n|:-|:-|:-|:-|:-|:-|\n|FCL[98]|RF-UC|-|13.16|24.23|22.01|\n|SCL[100]|RF-UC|-|19.07|30.39|28.08|\n|**LoTR**|RF-UC|-|**21.95**|**32.81**|**30.62**|\n|GEN-VLKT[17]|RF-UC|CLIP|21.36|32.91|30.56|\n|**LoTR**|RF-UC|CLIP|**25.97**|**34.93**|**33.17**|\n|FCL[98]|NF-UC|-|18.66|19.55|19.37|\n|SCL[100]|NF-UC|-|21.73|25.00|24.34|\n|**LoTR**|NF-UC|-|**23.68**|**25.23**|**24.87**|\n|GEN-VLKT[17]|NF-UC|CLIP|25.05|23.38|23.71|\n|**LoTR**|NF-UC|CLIP|**26.84**|**27.95**|**27.86**|\n|ATL[99]|UO|-|5.05|14.69|13.08|\n|FCL[98]|UO|-|0.00|13.71|11.43|\n|**LoTR**|UO|-|**8.41**|**21.35**|**18.48**|\n|GEN-VLKT[17]|UO|CLIP|10.51|28.92|25.63|\n|**LoTR**|UO|CLIP|**15.67**|**30.42**|**28.23**|\n|GEN-VLKT[17]|UV|CLIP|20.96|30.23|28.74|\n|**LoTR**|UV|CLIP|**24.57**|**31.88**|**30.77**|\n\nWe also give experiments under the regular HOI detection setup without using CLIP. Evidently, our LoTR continues to exhibit remarkable performance, up to $1.02$ mAP improvement over IF-HOI[17] which does not utilize CLIP.\n\n|Method|Backbone|VL Pretrain|Full|Rare|Non-Rare|\n|:-|:-|:-|:-|:-|:-|\n|UPT[15]|R50|-|31.66|25.94|33.36|\n|STIP[21]|R50|-|32.22|28.15|33.43|\n|ODM[106]|R50-FPN|-|31.65|24.95|33.65|\n|Iwin[107]|R50-FPN|-|32.03|27.62|34.14|\n|IF-HOI[17]|R50|-|33.51|30.30|34.46|\n|**LoTR**|R50|-|**34.53**|**31.12**|**35.38**|\n|SSRT[60]|R50|CLIP|30.36|25.42|31.83|\n|CATN[63]|R50|CLIP|31.71|24.82|33.77|\n|DOQ[62]|R50|CLIP|33.28|29.19|34.50|\n|GEN-VLKT[17]|R50|CLIP|33.75|29.25|35.10|\n|**LoTR**|R50|CLIP|**35.47**|**32.03**|**36.22**|\n\nThese experiments solidly demonstrate the effectiveness of our method under both zero-shot and regular setups, even compared with the meticulously tailored counterparts. \n\n---\n\n**Q3: Inferior performance on V-COCO compared to STIP[21].** \n\n**A3:** **First**, this may be attributed to the inherent characteristics of the V-COCO dataset, comprising merely 29 action categories, 80 object categories, and 263 kinds of human-object interactions. In light of this constrained diversity, the efficacy of our introduced logic-guided learning, which incorporates a large amount of affordances and proxemics properties into model reasoning, becomes especially modest. \n\n**Second**, STIP is a two-stage method that relies on a simple MLP to propose positive human-object pairs. Such a proposal network may achieve promising performance when the number of interaction classes is limited and exhibits inefficiency when applied to datasets with extensive interaction categories. Therefore, we experiment with the accuracy of the proposal network (percentage of successfully proposed positive human-object pairs) in STIP with the different number of interaction categories on HICO-DET:\n\n|#Category of Interaction|Accuracy|\n|:-:|:-|\n|20%|0.86|\n|40%|0.81 (-0.05)|\n|60%|0.73 (-0.08)|\n|80%|0.60 (-0.13)|\n|100%|0.45 (-0.15)|\n\nHere 20% means the training set only contains 0.2$\\times$600 classes of interactions in HICO-DET. As seen, the accuracy of the proposal network experiences an accelerated decline (*i.e.*, performance drops at a larger pace from 20% to 100%) when the number of interaction categories increases, validating our hypothesis above.\n\n---\n\n**Q4: Detailed elaboration to Table 5.**\n\n**A4:** In Table 5, we would like to probe the impact of different numbers of decoder layers and queries on the performance of LoTR. The **bolded** rows denote the setups that we finally adopted. As suggested, we add the statistics of params, FLOPs, and FPS for different configurations:\n\n|Layer(L)|Params|FLOPs|FPS|Full|Rare|Non-Rare|\n|:-|:-|:-|:-|:-|:-|:-|\n|2|45.4|82.11|18.62|34.61|30.72|35.54|\n|**3**|**49.8**|**89.65**|**16.84**|**35.47**|**32.03**|**36.22**|\n|4|54.3|97.27|14.75|35.37|31.96|36.09|\n|6|63.1|112.57|11.46|35.61|32.13|36.39|\n\n---\n\nThank you so much for your careful review. We will revise our manuscript according to your suggestive comments and please feel free to post your feedback if you have any further questions."}, {"review_id": "Ux7LTtNg6H", "reviewer": "Reviewer_AydW", "summary": "The paper addresses the problem of human-object interaction (HOI) detection. The authors introduce a new \"triplet-reasoning attention\" mechanism that enumerates all possible \"human-action-object\" triplets. Meanwhile, the authors propose a new logic-guided loss function, which enables an interaction detector with stronger reasoning capability. The new method (LoTR) is evaluated on HICO-DET and V-COCO, achieving competitive results.", "questions": "1. Acorrding to Line 180-182, the \"triplet reasoning attention\" actually enumerates all human-action-object combinations ($N_h \\times N_a \\times N_o$), in which most are negative proposals.\n2. The number of queries in this paper is $N_h \\times N_a \\times N_o = 32^3 = 32768$. However, in pervious approaches, the number of queries is normally less than 100. So the computational complexity should be reported.", "limitations": "Yes, the authors have addressed the limitations.", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 1, "contribution": 2, "strengths": "I believe that the authors may introduce some new points to the domain of HOI detection, especifically the logic-guided learning strategy and the proposed LoTR demonstrates excellent performance. However, the paper is really hard to follow due to the poor presentation.", "weaknesses": "- In Eq. 4, $\\mathbf{Q}^h \\in \\mathbb{R}^{N_h \\times 1 \\times D}$ and $\\mathbf{Q}^a \\in \\mathbb{R}^{1 \\times N_a  \\times D}$, why $\\mathbf{Q}^h + \\mathbf{Q}^a \\in \\mathbb{R}^{N_h \\times N_a}$\n- The $\\mathbf{X}$ are *inputs* (line 186) in Eq. 5 and *outputs* (line 191) in Eq. 6.  According to my understanding，the $\\mathbf{X}$ in Eq.5 are the outputs of \"triplet-reasoning attention\" while the $\\mathbf{X}$ in Eq.6 are the outputs of the decoder, i.e., the outputs of cross-attention (omitted residual connection and FFN).\n- In Eq. 7 and Eq. 8, what is $x(\\cdot)$ and what is x? is it a function or a variable? In Eq. 7, what is $M$? And it is really hard for me to understand L214-217. In line 213, $\\{ h_1, h_2,\\cdots, h_N\\}$ are infeasible interactions,  which yet seem to be reasonable interactions in line 217 (e.g., \"human-launch-boat\").\n- Eq. 11 and Eq. 12 need more detailed description. What is $M$ in Eq. 11 and what is $N$ is Eq. 12? Is $h_m$ different from $h_n$? What is $x_k$? The action-position loss, $L_{v,p} = (1-G_{v,p}) \\propto {1-s_k(v)[1-s_k(h_m)]}$, \nwhich decreases as $s_k(v)$ \nincreases and $s_k(h_m)$ decreases.  It can be intuitively interpreated that when an action *v* gets a high confidient score, then the interaction $h_m$ should get a low score. How to understand it?\n\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1: Clarify the dimension of $\\textbf{Q}^h$ + $\\textbf{Q}^a$.**\n\n**A1:** Sorry for this confusion. We omit a powerful feature named \"broadcasting\" which is widely used in Pytorch and Numpy.\nBroadcasting allows to perform element-wise operations between matrices of different shapes without explicitly duplicating the data. For example, when adding $\\textbf{Q}^h\\in\\mathbb{R}^{N_h \\times 1 \\times D}$ to $\\textbf{Q}^a\\in\\mathbb{R}^{1 \\times N_a \\times D}$, $\\textbf{Q}^h$ and $\\textbf{Q}^a$ will be firstly duplicated to $\\mathbb{R}^{N_h \\times N_a \\times D}$ along the first and the second dimensions, respectively. The related part will be updated to render a mathematically rigorous statement.\n\n---\n\n**Q2: Clarify $\\textbf{X}$ in Eq.5 and Eq.6.** \n\n**A2:** Apologies for this confusion. To improve readability, we reformulate these two equations as follows:\n\n$$\n\\textbf{X}'\\_{ij} =  \\textbf{W}^{v'}\\cdot\\sum\\_\\{n=1\\}^\\{N_a\\}\\text{softmax}{(\\textbf{F}^q\\_\\{in\\}\\cdot\\textbf{F}^k_\\{nj\\}/\\sqrt{\\textit{D}}})\\cdot\\textbf{F}^v\\_\\{inj\\}.  \\ \\ \\ \\ \\  \\text\\{(Eq.5)\\}\n$$ \n\n$$\n\\textbf{Y} = \\mathcal{D}^p(\\textbf{V}, \\textbf{Q}^h, \\textbf{Q}^a, \\textbf{Q}^o)\\in{\\mathbb{R}}^{N_h \\times N_o \\times D}, \\ \\ \\ \\ \\ \\ \\ \\ \\ \\ \\ \\ \\ \\ \\ \\ \\ \\ \\ \\ \\text\\{(Eq.6)\\}\n$$\n\nwhere $\\textbf{X}'$ denotes the output of  triplet-reasoning attention and the symbol $\\mathbf{Y}$ designates the ultimate output produced by the interaction decoder $\\mathcal{D}^p$.\n\n---\n\n**Q3.1: Clarify $x$ and $x(\\cdot)$ in Eq.7 and Eq.8.**\n\n**A3.1:** Sorry for this confusion. As specified in **L214**, the symbol $x$ here denotes a variable representing one human-object pair.  There is no $x(\\cdot)$ since Eq.7 and Eq.8 adhere rigorously to the syntax of standard first-order logic formula, where $x$ is closely bound with the universal quantifier $\\forall$ to compose $\\forall x$, denoting the assertion \"for all $x$\". \n\n**Q3.2: \"Human-launch-boat\" seems reasonable in L214-217.**\n\n**A3.2:** Though \"human-launch-boat\" seems feasible, it is essential to note that the positional cue is confined to \"object-above-human\" in **L216** (*i.e.*, $p$ is above), thereby rendering the feasibility of \"human-launch-boat\" unattainable within this context. The related part will be updated to make it more clear.\n\n---\n\n**Q4.1: Clarify $M$, $N$ and $h$ in Eq.11 and Eq.12.**\n\n**A4.1:** We sincerely apologize for this mistake that the human-object interaction sequence {$h_1, \\cdots, h_N$} in **L213** and **L255** should be {$h_1, \\cdots, h_M$}. Here $M$ refers to the number of infeasible interactions with pre-given action $v$ and position $p$, while $N$ refers to that with pre-given object $o$ and position $p$.\n\n**Q4.2: Clarify $x_k$ in Eq.11 and Eq.12.**\n\n**A4.2:** $x_k$ represents a human-object pair which is the instantiated value of variable $x$ and $K$ is the number of all human-object pairs.\n\n---\n\n**Q4.3: Explain $\\mathcal{L}_{v,p}$.**\n\n**A4.3:** Note {$h_1, \\cdots, h_M$} correspond to the set of **infeasible** interactions with the given verb $v$ (**L213**), which implies a **NEGATIVE** relationship between the verb $v$ and interactions {$h_1, \\cdots, h_M$}. In other words, as the score of $v$ increases, the scores of those infeasible interactions {$h_1, \\cdots, h_M$} should decrease.\n\n---\n\n**Q5: Most human-action-object combinations are negative.**\n\n**A5:** We follow the conventions that nearly all existing work [9-20,39-56] utilizes Focal loss for the classification of interactions which is capable of such an imbalanced positive-negative sample ratio. For instance, in HICO-DET, the average count of positive human-object pairs within an image is 4.65 and the possible combination number of our method is $16^3=4096$ (we only keep half of the human, object, and action queries as stated in **L195-196**) for $16^2=256$ human-object pairs (*i.e.*, 4.65 :256). In contrast, the detection task presents a positive-negative ratio of candidate boxes after filtering up to 1:2000 [ref1]. \n\nMoreover, our logic-guided reasoner learning (LRL) imposes additional constraints on the combinations of human-object pairs by considering exclusivity, *i.e.*, the interaction cannot be \"human-launch-horse\" when the action is \"launch\". This directs the model to focus on positive samples and is substantiated by ablative studies presented in Table 3 that the performance is notably enhanced from **34.32%** to **35.47%**.\n\n[ref1] Focal Loss for Dense Object Detection. ICCV 2017.\n\n---\n\n**Q6: Larger query number and computation efficiency.**\n\n**A6:** We give a short discussion on the computation efficiency of LoTR in **L356-362**, while a comprehensive comparison to existing work in terms of parameters, FLOPs, and inference speed is elaborated in the *Supplementary Materials* (**i.e.**, Table S1). The summarized results are presented below.\n\n|Method|Params|FLOPs|FPS|HICO-DET| \n|:-|:-|:-|:-|:-|\n|One-stages Detectors:|\n|PPDM|194.9|-|17.14|21.73| \n|HOTR|51.2|90.78|15.18|23.46| \n|QPIC|41.9|88.87|16.79|29.07|\n|CDN-S|42.1|-|15.54|31.78|\n|GEN-VLKs|42.8|86.74|18.69|33.75|\n|LoTR|49.8|89.65|16.84|**35.47**|\n\nIt can be seen that the params, FLOPs, and inference speed of our method align closely with existing one-stage work. The rationale behind this is discussed in **L356-362**: **i)** as stated in **L195-196**, we only keep **half** the number of human, object, and action queries by filtering the low-scoring ones before sending them into the interaction decoder, resulting maximum $16^3=4096$ human-action-object triplets, **ii)**, the layer number of Transformer decoder in LoTR is set to 3, which is the **smallest** among existing work, and **iii)**  triplet-reasoning attention introduces nearly no additional parameters. \n\n---\n\nThank you for providing such a comprehensive and detailed review. We are committed to reformulating our notation system and revising the methodology section in accordance with your valuable suggestions. Please do not hesitate to post comments if there are any further questions we can help.\n"}, {"review_id": "y1xQaEmFdU", "reviewer": "Reviewer_jaUK", "summary": "This paper introduces LoTR: Logic-Guided Transformer Reasoner for Human-Object Interaction (HOI) detection that adapts the self-attention mechanism in standard Transformers to infer feasible interactions over entities, guided by affordances and proxemics. ", "questions": "How to obtain the set of infeasible interactions (i.e., triplet ⟨human, action, object⟩) {h1, · · · , h_N} and {h1, · · · , h_M}?", "limitations": "none", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "none", "weaknesses": "1-The meanings of the notations in the paper are confusing. For instance, in Equation 2, the meaning of $W^{v\\'}$ is unclear, and there is no clarification provided in the text. It would be helpful if the authors could elaborate on the meaning of $X_i$. Since Equation 2 forms a significant contrast to the proposed triplet-reasoning attention (Equation 5), it should be thoroughly explained for improved readability.\n\n2-In Equation 5, the authors derive $X_{ij}$. It would be beneficial if they could provide a detailed explanation of how $X_{ij}$ differs from $X_i$.\n\n3-In Line 213, $h_N$ should be $h_M$. Furthermore, Equations 7 and 8 both have an extra right parenthesis.\n\n4-The method of designing a loss function using logical reasoning is interesting, but it seems rather complex. Could the authors use more intuitive language to describe $G_{v,p}$ and $G_{o,p}$?\n\n5-The introduction of logic-induced affordances and proxemics, in essence, imposes additional constraints on spatial positions within HOI. Could this lead to overfitting on these two datasets, thereby reducing the model's generalizability on a broader range of data? For example, I don't fully agree with the statement in Line 238, \"if there is a low possibility of a horse and the human box is above the object box, then the reasoning results should not be 'human-feed-horse.'\" This scenario is not entirely implausible as a person could feasibly feed a horse from atop it.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1: Clarify $\\textbf{W}^{v'}$ and $\\textbf{X}_i$ in Eq.2.**\n\n**A1:** Sorry for this confusion. Here $\\textbf{W}^{v'}$ is the weight of a linear layer that adapts the outputs of the attention mechanism. For $\\textbf{X}_i$ in Eq.2, it refers to the soft weight assigned to each **pre-constituted** human-object pair $i$, as the query $\\textbf{F}_i^q$ and key $\\textbf{F}_n^k$ are mapped from unified embeddings of human-object pairs (*i.e.*, $\\textbf{Q}^\\{h-o\\}$). For prior two-stage approaches, the **pre-constituted** human-object pairs are proposed by an MLP layer [21,39-53], and for one-stage methods, they are usually simultaneously constructed when predicting humans and objects [10-17].\n\n---\n\n**Q2: Clarify how $\\textbf{X}_{ij}$ in Eq.5 is different to $\\textbf{X}_i$ in Eq.2.**\n\n**A2:** For $\\textbf{X}\\_\\{ij\\}$ in Eq.5, it denotes the soft weight assigned to each possible human-object pair $ij$ composed by human $i$ and object $j$. The difference between $\\textbf{X}\\_\\{ij\\}$ in Eq.5 and $\\textbf{X}\\_i$ in Eq.2 is that the former encompasses arbitrary human-object interactions composed by given human $i$, action $n$, and object $j$ and reasons over them to compose the final output, while the latter pertains solely to human-object pairs $i$ that have already be established in advance. Evidently, the former delivers predictions in a compositional manner, which facilitates a comprehensive knowledge of different entities and the relations between them.\n\nTo improve readability, Eq.2, Eq.5, and Eq.6 are reformulated as follows:\n\n$$\n\\textbf{X}'\\_{i} =  \\textbf{W}^{v'}\\cdot\\sum\\_\\{n=1\\}^\\{N\\}\\text{softmax}{(\\textbf{F}^q\\_\\{i\\}\\cdot\\textbf{F}^k\\_\\{n\\}/\\sqrt{\\textit{D}}})\\cdot\\textbf{F}^v\\_\\{n\\},  \\ \\ \\ \\ \\ \\ \\ \\ \\ \\ \\text\\{(Eq.2)\\}\n$$ \n\n$$\n\\textbf{X}'\\_{ij} =  \\textbf{W}^{v'}\\cdot\\sum\\_\\{n=1\\}^\\{N_a\\}\\text{softmax}{(\\textbf{F}^q_\\{in\\}\\cdot\\textbf{F}^k\\_\\{nj\\}/\\sqrt{\\textit{D}}})\\cdot\\textbf{F}^v\\_\\{inj\\},  \\ \\ \\ \\ \\  \\text\\{(Eq.5)\\}\n$$ \n\n$$\n\\textbf{Y} = \\mathcal{D}^p(\\textbf{V}, \\textbf{Q}^h, \\textbf{Q}^a, \\textbf{Q}^o)\\in{\\mathbb{R}}^{N_h \\times N_o \\times D},  \\ \\ \\ \\  \\ \\ \\ \\ \\ \\ \\ \\ \\ \\ \\ \\ \\ \\ \\ \\  \\text\\{(Eq.6)\\}\n$$\n\nwhere $\\textbf{X}$ in Eq.2 and Eq.5 are replaced with $\\textbf{X}'$, and $\\textbf{X}$ in Eq.6 is replaced with $\\textbf{Y}$. $\\mathbf{Y}$ is the ultimate HOI prediction produced by the interaction decoder $\\mathcal{D}^p$.\n\n---\n\n**Q3: Typo.**\n\n**A3:** Sincerely thanks for pointing this out! We will correct them in the new version.\n\n---\n\n**Q4: Describe $\\mathcal{G}\\_\\{v,p\\}$ and $\\mathcal{G}\\_\\{o,p\\}$ intuitively.**\n\n**A4:** For $\\mathcal{G}\\_\\{v,p\\}$, it scores the satisfaction of predictions to rules defined in Eq.7. For example, given a high probability of action \"ride\" (*i.e.*, a high value of $s\\_k[v]$) and the position relationship is \"above\", if the probability of infeasible interactions (*e.g.*, \"human-feed-fish\") is high, then the value of $\\mathcal{G}\\_\\{v,p\\}$ would be low and $\\mathcal{L}\\_\\{v,p\\} = 1 - \\mathcal{G}\\_\\{v,p\\}$ should receive a large value so as to punish this prediction. $\\mathcal{G}\\_\\{o,p\\}$ is similar but it scores the satisfaction of predictions to Eq.8 with given position and objects such as \"horse\", \"fish\", *etc.*\n\n---\n\n**Q5: Overfitting brought by logic-guided constraints.**\n\n**A5:** **First**, the affordances and proxemics properties (*i.e.*, infeasible interactions with pre-given action, object, or position) are defined according to human knowledge, but not simply retrieved from the annotations of datasets. While the latter tends to overfit the datasets and ignore other possible interactions, the former is all-inclusive.\n\n**Second**, for each rule defined in Eq.7 and Eq.8, it is jointly measured by both action/object and position cues. This can reduce ambiguity. For example, though \"human-above-object\" can be \"person feed a horse from atop it\", **L238** also specifies that the possibility of a horse is **LOW** which makes \"human-feed-horse\" implausible.\n\n**Third**, as stated in **L253**, our logic-guided reasoner learning (LRL) only updates the parameters of the interaction decoder but not the entire network, so as to prevent overfitting.\n\n**Finally**, the zero-shot setup stands as a crucial approach for assessing the generalization capabilities of models [ref1]. As illustrated in Table 4, our logic-guided constraints yield notable performance improvements of **+1.96%**, **+2.41%**, and **+1.43%** across three different zero-shot setups. This demonstrates that the integration of logic-guided constraints not only averts overfitting but also amplifies the generalizability of our method.   \n\nThank you for raising such a good question. We will merge the above discussion into the main text. \n\n[ref1] Multitask Prompted Training Enables Zero-Shot Task Generalization. ICLR22\n\n---\n\n**Q6: Definition of infeasible interactions.**\n\n**A6:** We construct a large-scale knowledge base wherein the constituents are retrieved from VerbNet [ref1]. To illustrate:\n- First, when presented with a term like \"human-launch-XXX\", we utilize VerbNet to identify thematic roles associated with the corresponding verb class. These thematic roles signify the semantic relationships between the verb and the arguments. What we are interested in these roles is possible objects or entities that can be \"launched\". \n- Then, we generate a list of objects based on the thematic roles which can be used as suggestions or options for \"xxx\" in the context of \"human-launch-xxx\".\n- Finally, given all of the interactions (*i.e.*, \"human-action-object\") in HICO-DET or V-COCO, items with action other than \"launch\" or the objects not found within the aforementioned list are deemed infeasible. \n\n[ref1] VerbNet: A broad-coverage, comprehensive verb lexicon. University of Pennsylvania, 2005.\n\n---\n\nThank you for your valuable time and constructive feedback. We will revise the main text according to your review and the discussion above. It is welcome if there are any further questions or suggestions."}, {"review_id": "E3yjnaWzDz", "reviewer": "Reviewer_gPtK", "summary": "The paper introduces a novel approach to tackle the problem of human-object interaction detection. It presents two main contributions. Firstly, there is a revisited self-attention mechanism within the decoder that focuses on the interaction queries. This aspect has received limited attention in the field. Secondly, the paper introduces a logic-based reasoner that employs affordances and proxemics to filter out infeasible interactions. This reasoner replaces conventional hard-coded rule-based logic with differentiable operations. In my opinion, these technical advancements are original and are likely to be of interest to researchers in this field. Furthermore, experimental results on HICO-DET and V-COCO datasets demonstrate state-of-the-art performance.", "questions": "Please refer to the above section for further details. It would be greatly appreciated if the following questions could be addressed:\n\n- What are the potential benefits of integrating the reasoner into existing Transformer-based methods?\n\n- A more in-depth analysis explaining why the proposed method achieves improved results on unseen objects and unseen verbs.\n\n- Implementation details regarding the baseline method (as indicated in the first row of Table 3)?", "limitations": "The limitations have been discussed in the supplementary materials. The primary limitation arises from the computational complexity of the proposed method.", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "- Most Transformer-based HOI detection methods rely on the vanilla self-attention mechanism borrowed from DETR [59]. DIfferently, this paper introduces a reasonable enhancement to better adapt for the combination nature of human-object interactions. The idea is original and likely to be of interest to researchers in this field.\n\n- Some prior methods (e.g., [15][49]) have utilized rule-based logic to filter out infeasible human-object interactions, resulting in improved performance. This paper takes a further step by converting the rule-based logic into differentiable operations, which is very interesting.\n\n- The paper is generally well-written. Readers with a relevant background will have no difficulty following the paper.\n\n- The proposed method demonstrates significant improvements over existing approaches on both general and zero-shot settings.", "weaknesses": "There are a few minor weaknesses in the experimental section that should be addressed:\n\n- It remain unclear whether the proposed Transformer reasoner can be applied to common HOI detectors and contribute to performance improvements across different methods, or if it is specifically designed for the approach in this paper. To provide a comprehensive understanding, it would be valuable for the paper to include additional results such as GEN-VLK+reasoner, QPIC+reasoner, CDN+reasoner, and others, to showcase the potential benefits of integrating the reasoner into other existing methods.\n\n- In Section 4.2, there is a need for more in-depth analysis explaining why the proposed method can achieve better results on unseen objects and unseen verbs. Although an analysis on the reasoner is presented in L344-349, it remains unclear why TRA without the reasoner can also achieve state-of-the-art performance on the zero-shot setting, as shown in Table 4.\n\n- Regarding the baseline results in Table 3, it appears that the baseline method has already achieved better performance compared to many existing methods. The baseline does not seem to be a vanilla model with the standard Transformer encoder-decoder architecture. It would be helpful to address these concerns by providing additional implementation details.\n\nTypos:\n- L333: examine the efficiency of … -> examine the effect of …?\n\n- L45, L47: Fig.2 -> Fig.1\n\n- Eq.(7): $\\neg h_M(x)) \\rightarrow \\neg h_N(x)$", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1: Generalize to more existing Transformer-based HOI detectors.**\n\n**A1:** Sorry for this confusion. Our approach involves three distinct elements as input to the interaction decoder $D^p$: human, object, and action. In this context, the action item globally encompasses all possible actions occurring within the image, but unlike existing work that one single action is derived from a specific human-object pair. Therefore, the reasoner here cannot be directly adapted to existing work, but should add an additional decoding branch to predicate all potential actions. However, to properly address your concern, we give experiments on two approaches ( *i.e.*, QPIC[11] and GEN-VLKT[17]) by introducing an extra action decoder as well as our reasoner to them. The results are given below.\n\n| Method    | Backbone| Full    | Rare    | Non-Rare|\n|:----------|:--------|:--------|:--------|:--------|\n| QPIC[11]  | R50     | 29.07   |21.85    |31.23    |\n| QPIC + reasoner |R50| **32.23** (+3.16)   |**25.97** (+4.12) |**33.81** (+2.58) | \n| GEN-VLKT[17]|R50    | 33.75   |29.25    |35.10    | \n| GEN-VLKT + reasoner | R50 | **35.13** (+1.38) | **31.50** (+2.25) |**36.13** (+1.03) | \n\nWe do not heavily tune the hyper-parameters due to limited time, but there is still considerable improvement over these two methods which demonstrates the general effectiveness of our proposed logic-guided reasoner. \n\n---\n\n**Q2: Analysis on improvement under the zero-shot setup.**\n\n**A2:** For the improvement on zero-shot setups brought by Triplet-reasoning Attention (TRA), we provide discussions in **L76-77** and **L138-140** from the perspective of compositional generalization. Specifically, in our method, we break HOI detection into two parts: i) predicting all humans and objects and actions present in the image, and ii) employing TRA to reason over these three kinds of entities (**i.e.**, \"human\", \"object\" and \"action\") and compose the possible interactions (**i.e.**, \"human-action-object\"). Such a process entails a combination phase over three elements to deliver the ultimate prediction, but not directly fit the annotations. This improves the generalizability of the model and contributes to enhanced zero-shot HOI detection ability.\n\nFor the improvement brought by logic-guided reasoner learning (LRL), the affordances and proxemics properties are defined in an exclusive manner (i.e., the interactions can **NOT** be which categories with pre-given verb/object and position) in Eq.7 and Eq.8. As a result, even if the model has never encountered interactions like \"human-launch-boat\", the high possibility of action prediction \"launch\" excludes interaction predictions such as \"human-wash-boat\" and \"human-drive-boat\", eventually reinforcing the validity of the outcomes.\n\nTo render a more intuitive understanding of the improvement brought by each component, Table 4 is updated to:\n\n| TRA     |$\\mathcal{L}_{o,p}$|$\\mathcal{L}_{v,p}$| RF-UC | NF-UC |UO     | UV      |\n|:--------|:------------------|:------------------|:------|:------|:------|:--------|\n|         |                   |                   | 20.63 | 24.35 |9.64   | 20.23   | \n| &#10003;|                   |                   | 24.01 | 25.72 |13.26  | 23.14   |\n|         |      &#10003;     |                   | 25.22 | 26.45 |15.32  | 23.68   |\n|         |                   | &#10003;          | 25.34 | 26.57 |13.91  | 24.29   |\n| &#10003;|      &#10003;     | &#10003;          | 25.97 | 26.84 |15.67  | 24.57   |\n\nAs seen, TRA can boost the baseline methods up to **3.38** on RF-UC, **1.37** on NF-UC, **3.62** on UO, and **2.91** on UV, respectively. Such a solid improvement supports our motivation to address HOI detection from the perspective of compositional generalization.\n\nTable 4 and related parts will be updated.\n\n---\n\n**Q3: Implementation details of the baseline method.**\n\n**A3:** As specified in **A1** that the input to interaction decoder $D^p$ of LoTR contrasts significantly to the pipeline of existing work, we do not rely on the implementation of a specific approach but refer to a series of existing work. For instance, the architectural blueprints of human/object and interaction decoders adhere closely to CDN-S[13] which also merely adopts three Transformer decoding layers; the implementation of the action branch draws inspiration from [16]; the ultimate prediction follows GEN-VLKT[17] which delivers the class of interaction (*i.e.*, \"human-action-object\") but not a single action. The performance of the resultant baseline method is given below.\n\n| Method  | Backbone | Full | Rare | Non-Rare| AP$_\\{role\\}^\\{S1\\}$ | AP$_\\{role\\}^\\{S2\\}$\n|:--------|:---------|:-----|:-----|:--------|:--------|:--------|\n|baseline | R50      | 31.87| 26.14| 33.29   | 62.1    | 63.6    |\n|LoTR     | R50      | 35.47(**+3.70**)| 32.03(**+5.89**)| 36.22(**+2.93**)   | 64.4(**+2.3**)   | 65.6(**+2.0**)    |\n\nAs seen, despite the baseline having relatively higher performance compared to certain existing methods, LoTR still exhibits substantial improvements over the baseline (*i.e.*, **+3.7** on HICO-DET and **+2.15** on V-COCO), which is remarkable. \n\nTo ensure reproducibility, the implementation details will be enriched in *Supplementary Material* and our code shall be released upon acceptance.\n\n---\n\n**Q4: Typo.** \n\n**A4:** Sorry for these mistakes. We will revise them in the new version.\n\n---\n\n\nWe greatly appreciate you for your valuable time and constructive feedback. Hope our response has addressed your concerns and if you have any further questions, please feel free to raise them. We will do our utmost to give the clarification."}, {"review_id": "OBBFMnJDWQ", "reviewer": "Reviewer_uwRp", "summary": "This paper introduces a novel Human-Object Interaction (HOI) detection method called Logic-Guided Transformer Reasoner (LoTR). Unlike standard Transformer-based HOI detectors that accept pre-composed human-object pairs, LoTR leverages Transformers to infer plausible interactions among entities, thus enabling the creation of novel interactions. This is achieved by modifying the self-attention mechanism to reason over human-action-object triplets. The reasoning process is guided by two critical elements of HOI: affordances and proxemics, expressed in first-order logic and grounded into a continuous space. These adaptations enhance the learning of LoTR, leading to improved performance and zero-shot generalization capabilities. Evaluations on V-COCO and HICO-DET datasets demonstrate LoTR's superiority over existing methods under normal and zero-shot conditions.", "questions": "Please refer to the weaknesses.", "limitations": "I do not find any negative societal impact in this work.", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "+ A novel perspective for addressing problems in the Human-Object Interaction (HOI) domain, which is inspiring.\n\n+ An abundance of experimental evidence demonstrates that the method has yielded impressive results.\n\n+ The logic-guided learning method in section 3.3 of the paper is interesting and can provide assistance for the development of other computer vision tasks.\n", "weaknesses": "-  Although the method has great advantages and is thought-provoking in the HOI (Human-Object Interaction) field, its application seems somewhat limited. Can it be applied to other visual tasks? If so, I am really looking forward to such interesting things happening.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q: Applicability to visual tasks beyond HOI detection.** \n\n**A:** Thank you for your interest in the generability of LoTR. We firmly believe that the core principles of our logic-guided reasoning framework have the potential to be extended and applied to various computer vision tasks. This adaptability stems from:\n\n **First.** Our approach facilitates entity-centric reasoning so as to make use of the relationships between them. This contributes to holistic scene understanding [ref1] and a series of work in different disciplines can benefit from it. \n- For example, the consideration of relationships among objects of distinct semantic classes holds the potential for refining **image classification** and **object detection**. For instance, the co-occurrence of items like a mouse and laptop within a single image [ref2, ref3] can be harnessed.\n- Moreover, scenes and objects can be deconstructed into compositional elements (*e.g.*, the living room consists of several objects like a sofa, coffee table, and sofa made from cushions, armrests, *etc*.)[ref4]. A nuanced grasp of these mereological relationships between multiple parts shall significantly bolster **scene/object parsing** performance [ref5].\n- Additionally, in high-level scene understanding tasks such as **image caption** or **visual question answering (VQA)**, to solve questions like \"how many people are there near the bus?\" the model should not only identify \"people\" and \"car\", but also have to infer the relationships between them, reasoning whether the person is near the car, or far away from it [ref6].\n\n **Second.** We state world knowledge in first-order logic formulae and subsequently ground them into constraints towards guiding the model to generate more robust and logical sound predictions. This process serves as a vital complement to the entity-centric reasoning above. Since either the co-occurrence relation between objects or the topology of a specific scene/object is hard to directly retrieve by gradient optimization of training samples. Not to mention the extensive spectrum of human knowledge covering spatial arrangements, material properties, and even mathematical principles that are necessary for VQA. All of the above can be ground into constraints by our method and then assist the reasoning between different entities, *i.e.*, logic-guided reasoner learning. \n\n---\n\nTo demonstrate the discussion above, we experiment with our logic-guided reasoning framework on two tasks, *i.e.*, **weakly-supervised semantic segmentation** and **VQA**. \n\n\nFor **weakly-supervised semantic segmentation (WSSS)**, it has been proved that reasoning over different objects belonging to the same category can make up for the absence of detailed supervision by mining the co-patterns between them [ref7, ref8]. Here we follow [ref5] which takes three images as a group and then employs the graph neural network to conduct recursive reasoning for semantic understanding. Specifically, we replace GNN in [ref8] with a Transformer decoder where the self-attention is replaced with our triplet-reasoning attention (denoted as TRA). Moreover, we constrain the mereological relationships of objects in the COCO dataset. For example, \"knife\", \"fork\", \"spoon\" usually appear in the kitchen, and \"snowboard\", \"sports ball\", \"baseball bat\" are all sports equipment (denoted as LRL). The results on COCO are summarized below:\n\n\n|Method|mIoU|\n|:----------|:----------|\n|Baseline [ref8]|24.3|\n|+ TRA|26.9|\n|+ LRL|27.8|\n\nDue to the limited time, the performance reported above is the mIoU of the pseudo labels. In WSSS, these pseudo labels will next serve as the ground truth to train a segmentation network that is the same for all methods (*e.g.*, DeepLabV3). Thus the performance of pseudo labels can already reveal the effectiveness of the proposed method to some extent.\n\n\nIn the context of **VQA**, there have been continuous efforts towards leveraging knowledge to assist question answering by querying a pre-establish knowledge base [ref9]. Beyond the basic query operation, we further transform the knowledge into first-order logic formulae and use them to constrain the learning of models by our proposed logic-guided learning approach (denoted as LRL).\nThe results on the OK-VQA dataset are summarized below:\n\n|Method|accuracy v1.0|\n|:----------|:----------|\n|Baseline [ref9]|38.35|\n|+ LRL|39.42|\n\nFor fast implementation, we temporarily discard our triplet-reasoning attention here.\n\n\nGiven the above experiments on two different tasks, though the baselines we used may not be state-of-the-arts, the solid gain in performance can still demonstrate the general effectiveness of our method and its promising capacity to impart a positive impact on the general computer vision community. \n\n\n[ref1] Adopting Abstract Images for Semantic Scene Understanding. TPAMI 2014\n\n[ref2] Structure Inference Net: Object Detection Using Scene-Level Context and Instance-Level Relationships. CVPR 2018.\n\n[ref3] Joint binary classifier learning for ECOC-based multi-class classification. TPAMI 2015.\n\n[ref4] Vision GNN: An Image is Worth Graph of Nodes. NeurIPS 2022.\n\n[ref5] Towards Total Scene Understanding: Classification, Annotation, and Segmentation in an Automatic Framework. CVPR 2009.\n\n[ref6] Visual Query Answering by Entity-Attribute Graph Matching and Reasoning. CVPR2019.\n\n[ref7] Mining cross-image semantics for weakly supervised semantic segmentation. ECCV 2020.\n\n[ref8] Group-Wise Semantic Mining for Weakly Supervised Semantic Segmentation. AAAI 2021.\n\n[ref9] KRISP: Integrating Implicit and Symbolic Knowledge for Open-Domain Knowledge-Based VQA. CVPR 2021.\n\n---\n\nThanks again for your interest in the broader applications of our method. Hope our response has addressed your question and further suggestions are welcome. "}]

Author rebuttal:
# To all reviewers

We express our sincere gratitude to all reviewers for their valuable time and thorough assessment of our manuscript. In response, we have meticulously addressed each concern raised, providing point-to-point clarifications which shall be integrated into the new version of our manuscript.

We are gratified by the unanimous recognition from all reviewers that our logic-guided reasoning approach is interesting and novel, bringing significance within the context of the HOI detection task.

Foremost among the feedback is the concern about the writing of our manuscript, particularly with regard to the claim and elaboration of methodology. We acknowledge this and pledge to render a comprehensive revision of our article which will encompass rectifying overclaims, substantiating our conclusions through referencing literature, reformulating the symbol system for enhanced clarity, and providing a more detailed interpretation of the equations presented.

Additionally, in accordance with the above, Reviewer mDiH and gPtK also raise concerns about the evaluation of our method. We appreciate this and have refined both Table 1 and Table 2 to ensure a fair comparison with existing works. The analysis regarding params, FLOPS, and inference speed provided in *Supplementary Material* will be merged into Table 2, thereby affording an all-inclusive evaluation of our method. Moreover, we have given an analysis to the improvement in zero-shot setups brought by each component and tested the generability of our method by incorporating it into two existing methods.

Reviewer uwRp asks us about the application of logic-guided reasoning framework to other computer vision tasks. We are pleased to probe this as it takes the opportunity to prove the potential positive impact of our method to the general computer vision community. The results are also exciting, *i.e.*, substantial enhancements over the baseline performance in two tasks: Visual Question Answering (VQA) and weakly supervised semantic segmentation.


Meta-review: The AC and reviewers appreciate the authors rebuttal and discussion. All reviews, rebuttals and comments were carefully read and considered by the AC. The AC met with the senior AC to carefully review this paper. The reviewers agreed that the paper presents a new and interesting perspective for the HOI problem. For example gPtK wrote: "This paper takes a further step by converting the rule-based logic into differentiable operations, which is very interesting."  Reviewers found the paper mostly well written and the experimental section detailed and informative. Due to this the AC recommends that this paper be accepted to NeurIPS. 

However a number of the reviewers found some equations and details about the method to be confusing. Through the rebuttal process most of these issues have been clarified but the AC asks that the authors be sure to include these fixes in the final camera ready version of the paper to insure it maintains the NeurIPs bar for quality and correctness.
Decision: Accept (poster)