SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Lulei Li $^{1}$ , Jianan Wei $^{2}$ , Wenguan Wang $^{2*}$ , Yi Yang $^{2}$
$^{1}$ ReLER, AAII, University of Technology Sydney  $^{2}$ CCAI, Zhejiang University
https://github.com/weijianan1/LogicHOI
The interaction decoder utilized in prevalent Transformer-based HOI detectors typically accepts pre-composed human-object pairs as inputs. Though achieving remarkable performance, such paradigm lacks feasibility and cannot explore novel combinations over entities during decoding. We present LOGICHOI, a new HOI detector that leverages neural-logic reasoning and Transformer to infer feasible interactions between entities. Specifically, we modify the self-attention mechanism in vanilla Transformer, enabling it to reason over the  $\langle$  human, action, object  $\rangle$  triplet and constitute novel interactions. Meanwhile, such reasoning process is guided by two crucial properties for understanding HOI: affordances (the potential actions an object can facilitate) and proxemics (the spatial relations between humans and objects). We formulate these two properties in first-order logic and ground them into continuous space to constrain the learning process of our approach, leading to improved performance and zero-shot generalization capabilities. We evaluate LOGICHOI on V-COCO and HICO-DET under both normal and zero-shot setups, achieving significant improvements over existing methods.
The main purpose of human-object interaction (HOI) detection is to interpret the intricate relationships between human and other objects within a given scene[1]. Rather than traditional visual perception tasks that focus on the recognition of objects or individual actions, HOI detection places a greater emphasis on reasoning over entities [2], and can thus benefit a wide range of scene understanding tasks, including image synthesis [3], visual question answering [4, 5], and caption generation [6, 7], etc.
Current top-leading HOI detection methods typically adopt a Transformer [8]-based architecture, wherein the ultimate predictions are delivered by an interaction decoder. Such decoder takes consolidated embeddings of predetermined human-object pairs [9-20] as inputs, and it solely needs to infer the action or interaction categories of given entity pairs. Though achieving remarkable performance over CNN-based work, such paradigm suffers from several limitations: first, the principal focus of them is to optimize the samples with known concepts, ignoring a large number of feasible combinations that were never encountered within the training dataset, resulting in poor zero-shot generalization ability [21]. Second, the human-object pairs are usually proposed by a simple MLP layer [2, 22-37] or simultaneously constructed when predicting humans and objects [10-17], without explicit modeling of the complex relationships among subjects, objects, and the ongoing interactions that happened between them. Third, existing methods lack the reasoning ability, caused by ignoring the key nature of HOI detection, i.e., the interaction should be mined between entities, but not pre-given pairs.
In light of the above, we propose to solve HOI detection via the integration of Transformer and logic-induced HOI relation learning, resulting in LOGICHOI, which enjoys the advantage of robust distributed representation of entities as well as principled symbolic reasoning [38, 39]. Though
*Corresponding Author: Wenguan Wang.
Figure 1: Left: self-attention aggregates information across pre-composed interaction (□) queries. Middle: in contrast, our proposed triplet-reasoning attention traverses over human (■), action (■), and object (■) queries to propose plausible interactions. Right: logic-induced affordances and proxemics knowledge learning.
it was claimed that Seq2Seq models (e.g., Transformer) are inefficient to tackle visual reasoning problem[40, 41], models based on Transformer with improved architectures and training strategies have already been applied to tasks that require strong reasoning ability such as Sodomu [42], Raven's Progressive Matrices [43], compositional semantic parsing [44], spatial relation recognition [45], to name a few, and achieve remarkable improvements. Given the above success, we would like to argue that "Transformers are non-trivial symbolic reasoners, with only slight architecture changes and appropriate learning guidances", where the later imposes additional constraints for the learning and reasoning of modification. Specifically, for architecture changes, we modify the attention mechanism in interaction decoder that sequentially associates each counterpart of the current query individually (Fig. 1: left), but encourage it to operate in a triplet manner where states are updated by combining \(\langle\) human, action, object \rangle\) three elements together, leading to triplet-reasoning attention (Fig. 1: middle). This allows our model to directly reason over entity and the interaction they potentially constituted, therefore enhancing the ability to capture the complex interplay between humans and objects. To appropriately guide the learning process of such triplet reasoning in Transformer, we explore two HOI relations, affordances and proxemics. The former refers to that an object facilitates only a partial number of interactions, and objects allowing for executing a particular action is predetermined. For instance, the human observation of a kite may give rise to imagined interactions such as launch or fly, while other actions such as repair, throw are not considered plausible within this context. In contrast, proxemics concerns the spatial relationships between humans and objects, e.g., when a human is positioned below something, the candidate actions are restricted to airplane, kite, etc. This two kind of properties are stated in first-order logical formulae and serve as optimization objectives of the outputs of Transformer. After multiple layers of reasoning, the results are expected to adhere to the aforementioned semantics and spatial knowledge, thereby compelling the model to explore and learn the reciprocal relations between objects and actions, eventually producing more robust and logical sound predictions. The integration of logic-guided knowledge learning serves as a valuable complement to triplet-reasoning attention, as it constrains triplet-reasoning attention to focus on rule-satisfied triplets and discard unpractical combinations, enabling more effective and efficient learning, as well as faster convergence.
By enabling reasoning over entities in Transformer and explicitly accommodating the goal of promoting such ability through logic-induced learning, our method holds several appealing facets: first, accompanying reasoning-oriented architectural design, we embed affordances and proxemics knowledge into Transformer in a logic-induced manner. In this way, our approach is simultaneously a continuous neural computer and a discrete symbolic reasoner (albeit only implicitly), which meets the formulation of human cognition[46, 47]. Second, compared to neuro-symbolic methods solely driven by loss constraints [48-50], which are prone to be over-fitting on supervised learning [42] and disregard reasoning at inference stage, we supplement it with task-specific architecture changes to facilitate more flexible inference (i.e., triplet-reasoning attention). Additionally, the constraints are tailored to guide the learning of interaction decoders rather than the entire model, to prevent "cheating". Third, our method does not rely on any discrete symbolic reasoners (e.g., MLN[51] or ProbLog [52]) which increases the complexity of the model and cannot be jointly optimized with neural networks. Fourth, entities fed into Transformer are composed into interactions by the model automatically, such a compositional manner contributes to improved zero-shot generalization ability.
To the best of our knowledge, we are the first that leverages Transformer as the interaction reasoner. To comprehensively evaluate our method, we experiment it on two gold-standard HOI datasets (i.e., V-COCO[53] and HICO-DET[54]), where we achieve  $35.47\%$  and  $65.0\%$  overall mAP score, setting new state-of-the-arts. We also study the performance under the zero-shot setup from four different perspectives. As expected, our algorithm consistently delivers remarkable improvements, up to  $+5.16\%$  mAP under the unseen object setup, outperforming all competitors by a large margin.
HOI Detection. Early CNN-based solutions for HOI detection can be broadly classified into two paradigms: two-stage and one-stage. The two-stage methods [2, 23-37, 55, 56] first detect entities by leveraging off-the-shelf detectors (e.g., Faster R-CNN[57]) and then determine the dense relationships among all possible human-object pairs. Though effective, they suffer from expensive computation due to the sequential inference architecture [17], and are highly dependent on prior detection results. In contrast, the one-stage methods [58-61] jointly detect human-object pairs and classify the interactions in an end-to-end manner by associating humans and objects with predefined anchors, which can be union boxes [58, 61] or interaction points [59, 60]. Despite featuring fast inference, they heavily rely on hand-crafted post-processing to associate interactions with object detection results [10]. Inspired by the advance of DETR [62], recent work [9-20] typically adopts a Transformer-based architecture to predict interactions between humans and objects, which eliminates complicated post-processing and generally demonstrates better speed-accuracy trade-off. To learn more effective HOI representations, several studies [17, 19, 63-66] also seek to transfer knowledge from visual-linguistic pre-trained models (e.g., CLIP[67]), which also enhances the zero-shot discovery capacity of models.
Although promising results have been achieved, they typically directly feed the pre-composed union representation of human-object pairs to the Transformer to get the final prediction, lacking reasoning over different entities to formulate novel combinations, therefore struggling with long-tailed distribution and adapting to unseen interactions. In our method, we handle HOI detection by triplet-reasoning within Transformer, where the inputs are entities. Such reasoning process is also guided by logic-induced affordances and proxemics properties learning with respect to the semantic and spatial rules, so as to discard unfeasible HOI combinations, and enhance zero-shot generalization abilities.
Neural-Symbolic Computing (NSC) is a burgeoning research area that seeks to seamlessly integrate the symbolic and statistical paradigms of artificial intelligence, while effectively inheriting the desirable characteristics of both [39]. Although the roots of NSC can be traced back to the seminal work of McCulloch and Pitts in 1943 [68], it did not gain a systematic study until the 2000s, when a renewed interest in combining neural networks with symbolic reasoning emerged [51, 69-73]. These early methods often reveal limitations when attempting to handle large-scale and noisy data [74], as the effectiveness is impeded by highly hand-crafted rules and architectures. Recently, driven by both theoretical and practical perspectives, NSC has garnered increasing attention in holding the potential to model human cognition [38, 47, 75], combine modern numerical connectionist with logic reasoning over abstract knowledge [48, 76-81], so as to address the challenges of data efficiency [82, 83], explainability [84-87], and compositional generalization [88, 89] in purely data-driven AI systems.
In this work, we address neuro-symbolic computing from two perspectives, first, we employ Transformer as the symbolic reasoner but with slight architectural changes, i.e., modifying the self-attention to triplet-reasoning attention and the inputs are changed from interaction pairs to human, action, and object entities, so as to empower the Transformer with strong reasoning ability for handling such relation interpretation task. Second, we condense affordances and proxemics properties into logical rules, which are further served to constrain the learning and reasoning of the aforementioned Transformer reasoner, making the optimization goal less ambiguous and knowledge-informed.
Compositional Generalization, which pertains to the ability to understand and generate a potentially boundless range of novel conceptual structures comprised of similar constituents[90], has long been thought to be the cornerstone of human intelligence[91]. For example, human can grasp the meaning of dax twice or dax and sing by learning the term dax[40], which allows for strong generalizations from limited data. In natural language processing, several efforts[92-100] have been made to endow neural networks with this kind of zero-shot generalization ability. Notably, the task proposed in [40], referred to as SCAN, involves translating commands presented in simplified natural language (e.g., dax twice) into a sequence of navigation actions (e.g., I_DAX, I_DAX). Active investigations into visual compositional learning also undergo in the fields of image caption[6, 7, 101, 102] and visual question answering[4, 5, 103, 104]. For instance, to effectively and explicitly ground entities, [102] first creates a template with slots for images and then fills them with objects proposed by open-set detectors.
Though there has already been work [21, 31, 32, 105-107] rethinking HOI detection from the perspective of compositional learning, they are, i) restricted to scenes with single object [105], ii) utilizing compositionality for data augmentation [31, 106] or representation learning [32], without generalization during learning nor relation reasoning. A key difference in our work is that our method
arbitrarily constitute interactions as final predictions during decoding. This compositional learning and inference manner benefit the generalization to unseen interactions. Moreover, the affordances and proxemics properties can be combined, i.e., it is natural for us to infer that when a human is above a horse, the viable interactions options can only be sit_on and ride.
In this section, we first review Transformer-based HOI detection methods and the self-attention mechanism (§3.1). Then, we elaborate the pipeline of our method and the proposed triplet-reasoning attention blocks (§3.2), which is guided by the logic-induced learning approach utilizing both affordances and proxemics properties (§3.3). Finally, we provide the implementation details (§3.4).
3.1 Preliminary: Transformer-based HOI Detection
Enlightened by the success of DETR [62], recent state-of-the-arts [9-20] for HOI detection typically adopt the encoder-decoder architecture based on Transformer. The key motivation shared across all of the above methods is that Transformer can effectively capture the long range dependencies and exploit the contextual relationships between human-object pairs[10, 12] by means of self-attention. Specifically, an interaction or action decoder is adopted, of which the query, key, value embeddings  $\pmb{F}^q, \pmb{F}^k, \pmb{F}^v \in \mathbb{R}^{N \times D}$  are constructed from the unified embedding of human-object pair  $Q^{h-o}$  by:
$$
\boldsymbol {F} ^ {q} = \left(\boldsymbol {X} + \boldsymbol {Q} ^ {h - o}\right) \cdot \boldsymbol {W} ^ {q}, \quad \boldsymbol {F} ^ {k} = \left(\boldsymbol {X} + \boldsymbol {Q} ^ {h - o}\right) \cdot \boldsymbol {W} ^ {k}, \quad \boldsymbol {F} ^ {v} = \left(\boldsymbol {X} + \boldsymbol {Q} ^ {h - o}\right) \cdot \boldsymbol {W} ^ {v}, \tag {1}
$$
where  $\mathbf{X}$  is the input matrix,  $\mathbf{W}^q, \mathbf{W}^k, \mathbf{W}^v \in \mathbb{R}^{D \times D}$  are parameter matrices and  $Q^{h-o}$  can be derived from the feature of union bounding box of human-object[22] or simply concatenating the embeddings of human and object together[17]. Then  $\mathbf{X}$  is updated through a self-attention layer by:
$$
\boldsymbol {X} _ {i} ^ {\prime} = \boldsymbol {W} ^ {v ^ {\prime}} \cdot \sum_ {n = 1} ^ {N} \operatorname {s o f t m a x} \left(\boldsymbol {F} _ {i} ^ {q} \cdot \boldsymbol {F} _ {n} ^ {k} / \sqrt {D}\right) \cdot \boldsymbol {F} _ {n} ^ {v}. \tag {2}
$$
Here we adopt the single-head variant for simplification. Note that under this scheme, the attention is imposed over action or interaction embeddings, which has already been formulated before being fed into the action or interaction decoder. This raises two concerns i) may discard positive human-object pair and ii) cannot present novel combinations over entities during decoding.
3.2 HOI Detection via Triplet-Reasoning Attention
In contrast to the above, we aim to facilitate the attention over three key elements to formulate an interaction (i.e., human, action, object, therefore referring to triplet-reasoning attention), by leveraging the Transformer architecture. The feasible  $\langle$  human, action, object  $\rangle$  tuples are combined and filtered through the layer-wise inference within the Transformer. Towards this goal, we first adopt a visual encoder which consists of a CNN backbone and a Transformer encoder  $\mathcal{E}$  to extract visual features  $\mathbf{V}$ . Then, the learnable human queries  $\mathbf{Q}^h\in \mathbb{R}^{N_h\times D}$ , action queries  $\mathbf{Q}^a\in \mathbb{R}^{N_a\times D}$ , and object queries  $\mathbf{Q}^o\in \mathbb{R}^{N_o\times D}$  are fed into three parallel Transformer decoders  $\mathcal{D}^h$ ,  $\mathcal{D}^a$ ,  $\mathcal{D}^o$  to get the human, action, and object embeddings respectively by:
$$
\boldsymbol {Q} ^ {h} = \mathcal {D} ^ {h} (\boldsymbol {V}, \boldsymbol {Q} ^ {h}), \quad \boldsymbol {Q} ^ {a} = \mathcal {D} ^ {a} (\boldsymbol {V}, \boldsymbol {Q} ^ {a}), \quad \boldsymbol {Q} ^ {o} = \mathcal {D} ^ {o} (\boldsymbol {V}, \boldsymbol {Q} ^ {o}). \tag {3}
$$
Here,  $N_{h} = N_{a} = N_{o}$  and the superscripts are kept for improved clarity. All of the three query embeddings are then processed by linear layers to get the final predictions. For human and object queries, we supervise it with the class and bounding box annotations, while for the action queries, we only supervise them with image-level categories, i.e., what kinds of actions are happened in this image. After that, we adopt an interaction decoder  $\mathcal{D}^p$  composed by multiple Transformer layers in which the self-attention is replaced by our proposed triplet-reasoning attention, so as to empower Transformer with the reasoning ability. Specifically, in contrast to Eq.1, given  $Q^{h}, Q^{a}, Q^{o}$ , the input query, key, value embeddings  $F^{q}, F^{k}, F^{v}$  for triplet-reasoning attention are computed as:
$$
\boldsymbol {F} ^ {q} = \left(\boldsymbol {X} + \boldsymbol {Q} ^ {h} + \boldsymbol {Q} ^ {a}\right) \cdot \boldsymbol {W} ^ {q} \in \mathbb {R} ^ {N _ {h} \times N _ {a} \times D},
$$
$$
\boldsymbol {F} ^ {k} = \left(\boldsymbol {X} + \boldsymbol {Q} ^ {a} + \boldsymbol {Q} ^ {o}\right) \cdot \boldsymbol {W} ^ {k} \in \mathbb {R} ^ {N _ {a} \times N _ {o} \times D}, \tag {4}
$$
$$
\boldsymbol {F} ^ {v} = \boldsymbol {W} _ {h} ^ {v} \cdot (\boldsymbol {X} + \boldsymbol {Q} ^ {h} + \boldsymbol {Q} ^ {a}) \odot (\boldsymbol {X} + \boldsymbol {Q} ^ {a} + \boldsymbol {Q} ^ {o}) \cdot \boldsymbol {W} _ {o} ^ {v} \in \mathbb {R} ^ {N _ {h} \times N _ {a} \times N _ {o} \times D},
$$
Figure 2: Overview of LOGICHOI. We first retrieve human, action, and object queries by  $\mathcal{D}^h$ ,  $\mathcal{D}^a$ , and  $\mathcal{D}^o$ , respectively. Then  $\mathcal{D}^p$  take them as input, reasoning over entities and combining potential interaction triplets. Finally, such process is guided by affordances and proxemics properties, to be more efficient and knowledge-informed.
where  $\odot$  is the element-wise production. Note that we omit the dimension expanding operation for better visual presentation. Concretely, for  $F^q = (X + Q^h + Q^a) \cdot W^q$ , the human queries  $Q^h \in \mathbb{R}^{N_h \times D}$  and action queries  $Q^a \in \mathbb{R}^{N_a \times D}$  are expanded to  $\mathbb{R}^{N_h \times 1 \times D}$  and  $\mathbb{R}^{1 \times N_a \times D}$ , respectively. In this manner,  $Q^h + Q^a$  associates each human and action entity, resulting in  $N_h \times N_a$  human-action pairs in total.  $N_a \times N_o$  viable action-object pairs are risen by  $Q^a + Q^o$  in the same way. For the value embedding  $F^v$ , it encodes the representation of all  $N_h \times N_a \times N_o$  potential interactions. Given a specific element in  $F^v$ , for instance,  $F_{inj}^v$ , it is composed from  $F_{in}^q$  and  $F_{nj}^k$ , corresponding to a feasible interaction of embeddings  $Q_i^h$ ,  $Q_n^a$ , and  $Q_j^o$ . Then each element in inputs  $X$  is updated by:
$$
\boldsymbol {X} _ {i j} ^ {\prime} = \boldsymbol {W} ^ {v ^ {\prime}} \cdot \sum_ {n = 1} ^ {N _ {a}} \operatorname {s o f t m a x} \left(\boldsymbol {F} _ {i n} ^ {q} \cdot \boldsymbol {F} _ {n j} ^ {k} / \sqrt {D}\right) \cdot \boldsymbol {F} _ {i n j} ^ {v}, \tag {5}
$$
where  $X'$  denotes the output of triplet-reasoning attention. In contrast to self-attention (cf., Eq.2), our proposed triplet-reasoning attention (cf., Eq.5) stretches edges between every human-action and action-object pairs sharing the identical action query. By aggregating information from the relation between human-action and action-object, it learns to capture the feasibility of tuple  $\langle \text{human}, \text{action}, \text{object} \rangle$  in a compositional learning manner, simultaneously facilitating reasoning over entities. The final output  $Y$  of  $\mathcal{D}^p$  is given by:
$$
\boldsymbol {Y} = \mathcal {D} ^ {p} \left(\boldsymbol {V}, \boldsymbol {Q} ^ {h}, \boldsymbol {Q} ^ {a}, \boldsymbol {Q} ^ {o}\right) \in \mathbb {R} ^ {N _ {h} \times N _ {o} \times D}, \tag {6}
$$
which delivers the interaction prediction for  $N_{h} \times N_{o}$  human-object pairs in total. We set  $N_{h}, N_{a}, N_{o}$  to a relatively small number (e.g., 32) which is enough to capture the entities in a single image, and larger number of queries will exacerbate the imbalance between positive and negative samples. Additionally, for efficiency, we keep only half the number of human, object and action queries by filtering the low-scoring ones before sending them into the interaction decoder  $\mathcal{D}^{p}$ .
3.3 Logic-Guided Learning for Transformer-based Reasoner
In this section, we aim to guide the learning and reasoning of LOGICHOI with the affordances and proxemics properties. Though there has already been several works concerning about these two properties[15, 33, 63], they typically analyze affordances from the statistical perspective, i.e., computing the distribution of co-occurrence of actions and objects so as to reformulate the predictions [33], simply integrate positional encodings into network features [15, 22], or proposing a two-path feature generator [63] which introduces additional parameters. In contrast, we implement them from the perspective that the constrained subset is the logical consequence of pre-given objects or actions.
Concretely, we first state these two kinds of properties as logical formulas, and then ground them into continuous space to instruct the learning and reasoning of our Transformer reasoner (i.e., interaction decoder  $\mathcal{D}^p$ ). For proxemics, we define five relative positional relationships with human as the reference frame, which are above (e.g., kite above human), below (e.g., skateboard below human), around (e.g., giraffe around human), within (e.g., handbag within human), containing (e.g., bus containing human). To make the Transformer reasoner spatiality-aware, the human and object embeddings retrieved from Eq.3 are concatenated with sinusoidal positional encodings generated from predicted bounding boxes. Then, given action  $v$  and position relationship  $p$ , the set of infeasible interactions (i.e., triplet  $\langle \text{human}, \text{action}, \text{object} \rangle$ )  $\{h_1, \dots, h_M\}$  can be derived:
$$
\forall x (v (x) \wedge p (x) \rightarrow \neg h _ {1} (x) \wedge \neg h _ {2} (x) \wedge \dots \wedge \neg h _ {M} (x)), \tag {7}
$$