ART: rule bAsed futuRe-inference deducTion
Authors: Mengze Li
Tianqi Zhao
Bai Jionghao
Baoyi He
Jiaxu Miao
Wei Ji
Zheqi Lv
Zhou Zhao
Shengyu Zhang
Wenqiao Zhang
Fei Wu
Venue: EMNLP 2023 Main (2023)
Keywords: cross-modal
deductive reasoning
deep learning

Abstract: Deductive reasoning is a crucial cognitive ability of humanity, allowing us to derive valid conclusions from premises and observations. However, existing works mainly focus on language-based premises and generally neglect deductive reasoning from visual observations. In this work, we introduce rule bAsed futuRe-inference deducTion (ART), which aims at deducing the correct future event based on the visual phenomenon (a video) and the rule-based premises, along with an explanation of the reasoning process. To advance this field, we construct a large-scale densely annotated dataset (Video-ART), where the premises, future event candidates, the reasoning process explanation, and auxiliary commonsense knowledge (e.g., actions and appearance) are annotated by annotators. Upon Video-ART, we develop a strong baseline named ARTNet. In essence, guided by commonsense knowledge, ARTNet learns to identify the target video character and perceives its visual clues related to the future event. Then, ARTNet rigorously applies the given premises to conduct reasoning from the identified information to future events, through a non-parametric rule reasoning network and a reasoning-path review module. Empirical studies validate the rationality of ARTNet in deductive reasoning upon visual observations and the effectiveness over existing works.

ART: rule bAsed futuRe-inference deducTion
Mengze Li', Tianqi Zhao', Jionghao Bai', Baoyi He', Jiaxu Miao', Wei Ji', Zheqi Lv Zhou Zhao Shengyu Zhang', Wenqiao Zhang", Fei Wu 'Zhejiang University SShanghai Institute for Advanced Study of Zhejiang University 3 SShanghai AT Laboratory
Abstract
Observed Phenomenn (O'
Deductive reasoning a crucial cognitive abil- ity of humanity, allowing us to derive valid con- clusions from premises and observations. How- ever, existing works mainly focus on language- based premises and generally neglect deduc- tive reasoning from visual observations. In this work, introduce rule bAsed futuRe- inference deducTion (ART), which aims at de- ducing the correct future event based on the vi- sual phenomenon video) and the rule-based premises, along with an explanation of the rea- soning process. To advance this field, we con- struct large-scale densely annotated dataset (Video-ART), where the premises, future event candidates, the reasoning process explanation, and auxiliary commonsense knowledge (e.g., actions and appearance) are annotated by anno- tators. Upon Video-ART, develop strong baseline named ARTNet. In essence, guided by commonsense knowledge, ARTNet learns identify the target video character and per- ceives its visual clues related to the future event. Then, ARTNet rigorously applies the given premises to conduct reasoning from the iden- tified information to future events, through a non-parametric rule reasoning network and reasoning path review module. Empirical stud- ies validate the rationality of ARTNet in deduc- tive reasoning upon visual observations and the effectiveness over existing works.
Rule Set (R):
L: the oloor spics somssone the mase will
waiking
Kale specsig bag wodlied por dusk
Rale Sissonne
bag will Aag
Candidute Future Exents:
yelhyeeloe Tveer 1: The fndy with yelloe Aair ir pusting fie bask the destt (ito fer hog.
(R Cunase
Ceduce)
Sivent 2: The womer
dawen her-bag
3C
fody opoosed
Process for Ø:
Rezseming
Phenemeena
Olburvvd
Rute
Rale
Conec
Cone
Muke
Everr
Cha-- The jurice 4ck vvsued spes Eap
Figure 1: An illustration of the ART task. With the observed phenomena
the rule set R, the ART task aims reasoning the correct one from candidate future events and explaining the reasoning process based on the rule set
stone of human psychological functioning, serving as an indispensable aspect of our daily cognitive processes. For example, human beings possess the capability to utilize the given rule set (R) to de- duce future events (F) through the interpretation of observed phenomena (O). To illustrate:
Supposing that the rule R: the self- protection, the person will release the hot ob- ject, once burned. holds, and we observe O: a man holds a very hot teacup, it follows logi- cally that we anticipate F: he will release the cup.
1 Introduction
Deductive reasoning a systematic method that rigorously follows a set of explicitly given constraints (i.e,, rules) to deduce valid conclu- sions from empirical facts through logical infer- ences (Sanyal et al., 2022b). It represents a comne-
Under the premise that R: after getting home, my dad will definitely smoke to relieve anx- iery;, the observation O: dad returns home from work at night, should lead to the future event F: he will smoke.
First author. menggzli@zju.eduu.
"Coresponding author.
zhaozhou @ edu.edu..nn
zhang @ qju.edu.cr
wengiaozhang @ @rwj..ddd.c. edu.en
Despite deductive reasoning being acknowledged

---

fundamental cognitive competency of human- ity (Rips. 1994; Evans al., 1993), there is scarcity of investigations designing Al systems that are capable executing deductive reasoning in the multi-modal field.
closely associated with the visual information pre- sented; (2) The annotators provided the correct future events and rule-based explanation of the reasoning process. In addition, to enhance the AI system's deduction from visual semanties highly unstructured videos, which are composed of densely arranged pixels, we have carefully an- notated the commonsense knowledge of the target objects, including their appearance and related ac- tions.
To advance the research, we simulate the deduc- tive reasoning of human beings and propose a rule bAsed futuRe-inference deducTiion task (ART). Overall, in aligning with the established deduc- tive reasoning studies within the NLP community (Sanyal al., 2022b), ART should select the cor- rect textual fitture events (correct conclusions) from potential candidates, rigorously based on the given language rule set and the observed phenomena. As depicted Figure 1, the observations in ART are illustrated by videos, which is grounded in the fact that visual information has a profound impact on the human brain, accounting for an impressive 83% of all inputs (Rosenblum, 2011) 1). Upon the video observations, ART reasons on crucial ingredient of videos, i.e., human actions, and endeavors to derive the most accurate future event, constrained by the given rule set and the video observation. In addition to deducing future events, ART should provide explanations of the reasoning process by generating the rule chain. In formal terms, our ART task is close to the well-established video- language inference task (Liu al., 2020; Li et al., 2021), which assesses the accuracy of language descriptions in relation to input videos. Analyzing the difference between the two tasks can facili- tate the advancement of our new field. Compared with the video-language inference, our ART task has the following characteristics: (1) The task de- mands transparent explanation of the reasoning process; (2) In accordance with deductive reason- ing (Sanyal al., 2022b), the reasoning processes should be constrained by the rule sets (regardless of the perceived irrationality of provided rules) that are explicitly assigned to each individual sample, while relying on implicitly learned rule constraints that might come from other samples could cause mistakes.
To lay the groundwork for future research, we propose a strong baseline for the rule based future-inference deduction, named ARTNet. ART- Net mainly consists of three components, i.e., knowledge-guided target perception (KTP), non- parametric rule reasoning network (RRN), and rea- soning path review (RPR). KTP learns identify the target character and corresponding visual clues related to the upcoming event through multi-task lcarning and commonsense knowledge annotations such as actions and appearance. Inspired by tradi- tional graph-theoreti algorithms, RRN performs layer-by-layer reasoning through purpose-buil non-parane con-parametric rule reasoning network, uncover- ing the reasoning paths from the identified visual clues to potential future events. RRN offers two ad- vantages for the ART task over traditional models:
(1) RRN provides explanations of its rule-based reasoning process. (2) RRN avoids rote memoriza- tion of rules within the training data and ensures the rigorous application of the sample-specific rule set. Furthermore, the RPR module validates the semantic consistency between the rule reasoning paths uncovered by RRN, the video observations. and the future event descriptions.
Overall, the main contributions of this work are three-fold:
We propose the rule based future-inference deduction task, through imitating human cog- nition. To the best of our knowledge, this is an early exploration of deductive reasoning in the multi-modal domain.
To promote multi-modal deductive reasoning re- search and meet the demands of the ART task, we introduce dataset, named Video-ART, con- sisting of 23, 895 samples. Careful annotation was performed by annotators and verifiers with strong logical reasoning skills, who mainly focused on two key aspects: (1) They targeted to design the rule sets and the candidate future events that are
We construct a large-scale dataset Video-ART to promote multi-modal deductive reason- ing research. Video-ART consists of 23, 895 examples where dense annotations including the rule set, reasoning processes, and auxiliary commonsense knowledge are provided.
Please contact mengreli@zju..duu for dataset acquisi-
tion.

---

We contribute strong baseline, ARTNet, tai- lored for the ART task. Experimental results on the Video-ART dataset validate the effec- tiveness of ARTNet over the waae-offthhe-rrs
Category
Subeategory
Appearance
Gender, Hair Length. Age
Clothing
Length of Lower-bedy Clothing. Type of Lower-body Clothing. Type of Upper-body Clothing, Sleeve Length, 3 Other Outfits, 9 Colors of Upper- body Clothing. 9 Colors of Lower-body Clothing
2 Related Work
Video-Language Inference. As the development of the deep learning (Wu et al., 2020; Miao et al., 2021; Wu et ul., 2022; Ji et al., 2023b,c), the visual and language related tasks attracts more and more attentions (Wu et al., 2023; Li al., 2023a; Miao al., 2022; Ji et al., 2023a; Li al., 2022b,c). The video-language inference task aims judging the correctness of the textual conclusion, based on the video information and the language descrip- tion (Li et al., 2020; Tang et al., 2021; Chen and Kong, 2021; Zhang et al., 2019; Gokhale et al., 2022). (Liu et al., 2020) proposes carefully la- beled dataset for this task and introduces strong baseline to further develop this field. Based on this dataset, (Li al., 2021) designs new model based on the graph network and validate it.
Action
Intransitive Verb, Transitive Verb, Ob- ject
Sentiment
No Subcategories
Scene
No Subcategories
Table Statistics of commonsense knowledge types
for Video-ART.
video); (2) explain the logical chain leading to the final conclusion based on the rules. Taking into account the ART task characteristies, propose an exhaustively labeled large-scale dataset, named Video-ART.
Future Event Prediction. Our task is formally related to future event prediction, whose goal is to infer the future from known facts (Surís et al., 2021; Vondrick et al., 2016; Epstein and Vondrick, 2021). This field encompasses many tasks, such the future trajectory prediction (Kim et al., 2021; Chen al., 2022: Li et al., 2022a; Alahi al., 2016), the action prediction (Abu Farha et al., 2018; Kitani et al., 2012; Lan et al., 2014; Ryoo, 2011), and future object localization (Jia et al., 2022; Peri et al., 2022). Graph-based methods (Zhang et al., 2021, 2022) are often considered alternative solutions for such tasks.
3.1 Data Collection
We collect the videos in our dataset from two sources: (1) Parts of the video clips are manually intercepted from 80 American movies, including Broke Girls, Grey's s Anatomy, Mr. Bean, etc. These videos are of high quality, with rich character ac- tions and emotions, and rigorous plot logic. (2) Other videos carefully selected from the ex- isting datasets, Charades (Sigurdsson al., 2016) and TO-MAR (Li et al., 2023b), both of which are the human-center datasets. These videos consist of many different actors and scenes. The appearance, clothing, etc of the characters are richer.
Deductive Reasoning. Reasoning is impor- tant skill for human beings to understand the world (Byme, 1989; Rips, 1994) To promote the develop- ment of human-like reasoning Al systems, many researchers have invested in this field (Ebrahimi et al., 2021; Li et al., 2023b; Calimeri et al., 2021; Sanyal et al., 2022a). (Sanyal et al., 2022b) pro- poses deductive reasoning task in the field of NLP and presents a powerful model that outperforms previous methods.
Both sources of data have their own characteris- tics and combined together may provide relatively comprehensive testbed for the ART task. Some collected videos are not suitable for our ART task, such as videos with few actions or blurred videos in which key details cannot be clearly distinguished.
3.2 Data Annotation
With the collected videos, we rigorously design the ART task examples for each data and manually validate all examples. In addition, we annotate the commonsense knowledge for all video characteris- tics in detail to assist Al system training, including human appearance, clothing, actions, semantics, and scenes located.
3 Deductive Reasoning Dataset
Our rule bAsed futuRe-inference deducTiion task (ART) requires the AI system to (1) select the correct future event from the candidate events by reasoning on the rule set and the observation (a
Commonsense Knowledge Annotation. The

---

Video (actions)
50000
such as working, cleaning, cooking, etc. In addition, on average there are future events in each example. The average length of the videos is 21.5 seconds. Detailed statistics are shown in Figure 2. We show more statistical results in the appendix.
>=1]
Item Number
40000
22.69
9-13
30000
25.7%
15.0%
20000
10000
36.8%
89 9-12 12-15 >15
5-9
Sentencee (words)
Method
Figure 2: Statistics about the dataset: (1) the videos with different action numbers; (2) the future events with different word numbers.
We propose new task, rule bAsed futuRe- inference deduc'Tion (ART), and design a targeted model named ARTNet. According to the task char- acteristics, we contribute the non-parametrie rule reasoning module for ARTNet. In addition to the key reasoning module, the knowledge-guice per- ception module and the rechecking module are in- troduced to assist in the completion of the ART task.
annotated categories and subcategories of common- sense knowledge for each video characteristic are shown in Table 1. More annotation details are shown in the appendix.
ART Task Annotation. Our annotations for the ART task contain two aspects: (1) the rule set and the candidate future events for each video; (2) the labels indicating the correct future event and the complete rule chain as the explanation coming for the correct event. Specifically,, achieve rigor- ous labeling, doctoral and undergraduate students from the top 50 universities the world are respon- sible for annotation. Firstly, the annotators crop out appropriate video clips as video observation. In reference to the commonsense knowledge and the subsequent video content of the cropped video clips, the correct future event and the mased explanation of the reasoning process for this event (the rule chain) are annotated. Then, according to the correct event, the confusing items in both the candidate future events and the rule set are supple- mented.
Task Formulation. Given an observation (a video) V, rule (multiple rules) S No and candidate future events C = { described by the natural language, the ART task aims to rea- son out the correct future event and explain the reasoning process based on the rules. We define the model with the parameter for the ART task as M. Then, the training optimization function 8(.) of M is represented as:
{
VC,
=max E(c(C, M.c. (1)
6
where θ is learnable parameter. The function c(-)
generates the ground truth and the function M
outputs the model prediction. The function E(.) calculates the consistency of e(-) and M(.).
Validation. The verifiers with strong logical abilities are responsible for verifying the labeled examples. The examples not agreed by them relabeled discarded.
Rule Transformation. Before describing our ARTNet structure, we shed light on the intriguing transformation of rules within the ART task. The ART task revolves around predicting future events based on observed information, employing rules as the means of inference. These rules can be per- ceived as an intricate mapping of crucial informa- tion bridging two consecutive events. is impor- tant to note that the core driver of event progression for the target character lies in the changes in ac- tions. Hence, in our proposed baseline, we adopt approximation where rules represented mappings of actions. We employ the robust Stan- fordNLP toolkit (Manning et al., 2014) identify the key actions within each rule, which serve as the basis for future-inference deduction and explana- tion.
3.3 Dataset Analysis
Our dataset has the following characteristics: (1) Deductive Reasoning Orientated. The Video- ART dataset is strictly designed according to the ART task. It is the first deductive reasoning dataset in the multi-modal field collected from different human scenes. (2) Large-scale. The dataset con- sists of 23, 895 examples. Among them, 5, 922 examples are collected from movies and 17, 973 examples come from real-life scenarios. (3) Diver- sity. (1) The examples in the dataset are rich in scenarios, including residence, hospital, restaurant, etc. (2) The dataset covers a variety of activities

---

Step Knowledge-guidde Target Perception
Query Vectors
Frame Temme-Text Level Feature
Frame- Trame-Textl Level
Video-Toxt Level
Visual Observation:
Video-Text Level Feature
Transformer Encoder
Information Flow
Transformer Decoder
Visual Clue Retrieve
Future Event:
MLP
MLP
MLP
MLP
The lady with yellow hair is putting the book into the bag.
Semantic Soone
Semantic Scene
Appearance Cloching
(Piil Goto conns Corry bugg Spooi
R
Put book
Run to desk
Walk into rooel
Walk Run desk
into bag
Review Module
desk book
Correct Future Event
Open bag
Take wallet
Storage
Graph Construction
Rule Path Finding
Step Reasoning Path Review
Step Non-parametric Rule Reasoning
Figure 3: Overview of our ARTNet model for the ART task. It consists of three steps. Step I: Knowledge-guiide Target Perception, which focuses on the language-described person Step 2: Non-parametric Rule Reasoning, which constructs the rule graph (action graph), and finds the connected action path between the video actions and the future-event action with Dijkstra's algorithm. Step 3: Reasoning Path Review, which finally checks whether the future event, the video, and the found rule paths match determine the correctness of the future event.
video and identifies her key actions.
Model Pipeline. As shown in Figure 3, our ARTNet judges the correctness of candidate future events one by one based on the given rule S and the video V. chooses the correct item with the highest probability as the prediction result and outputs the rule-relied inference process as an explanation. Specifically, the whole process is divided into three steps: Step 1 (Section 4.1): By analyzing the input video V and the future event target perception module
future event F. We choose the with the highest probability from the candidate future events the final prediction result of the deductive reasoning task, ART. The corresponding rule path in Step 2 is viewed as the explanation for the prediction result. Step 3 is introduced in the appendix in detail.
4.1 Knowledge-guided Target Perception
Identifying the target character and the correspond- ing action knowledge related to the natural lan- guage future event F from the input video V is an essential step before the reasoning based on the rule set S (which is transformed into action chains in the preprocessing). Toward this target, leverage Human-annotail commonsense knowledge labels (human appearance, clothing, semantics, scene, and actions) the target vidco person described by the textual future event F to train the model with transformer-bast multi-task learning.
F. the knowledge-guine focuses the person described by the language event F and predicts her key visual action. Step 2 (Section 4.2) The non-paramen parameeriii rule reasoning network constructs the rule graph based on the action chains stored in the rule memory. Then, finds the connected rule paths between the future event action and the visual action from the video V. If no path is found, we judge the future event JF is wrong. Step 3 (Appendix): The review module of ARTNet reasons on the found connected rule paths and the cross-modal feature containing the semantics of the video V the future event F. The module outputs the correct probability Pe the
Specifically, the knowledge-guided visual per- ception module designed based on the trans- former architecture. The transformer encoder ex-
tracts the cross-modal feature F, from the video

---

V and the future event F. We define two types of query vectors to analyze the cross-modal feature Fe with the transformer decoder. It includes the frame-text level queries 2 (q))it) and the video-text level queries Q [ used an- alyze the cross-modal semantics, where Ny is the number of frames. The transformer decoder distin- guishes different types of query vectors according to the injected type embeddings. Then, reasons the corresponding features (frame-text level fea- tures F, (ry and video-text level features F, (f,)22) relying the query vectors (Q, and Q.). By analyzing the resulting features ,F, and F,), we predict all commonsense knowledge. We take the action knowledge predeition an ex- ample, the others are shown in appendix.
process represented as:
(A1 + A2),. (A. - Ar) -spit(AA A,).
(3)
Notably, for the action chain form of combinato- rial inference in the action chain set A, A, -> Ag. store two single-step relational maps. A - A and Aj struction. We construct the action graph ((A, by connecting all the single-step relational maps (A; + Aa A.1))..., (As ory, where U represents the edges between the ac- tions A in the graph. The construction process is formalized as:
Ak- (2) Graph Con-
+ Aiin) in the mem-
G(A, connect((A; + Aiti),
Action Knowledge Prediction. For action knowledge, we do predict it frame-by-frame like sentiments and scenes. multiple actions for the target person in each frame and the frame-by- ba--bby--feeeeeee prediction introduces too much burden to ARTNet, which leads to diffi- cult model training. Therefore, the model directly counts the actions contained in the video rather than each frame. The process of judging whether the i-th action exists in the video V with the video-level query f. is represented as:
(Ar-n-1 Altn)). (4) (3) Action Path Finding, Firstly, need find the starting nodes (A,) Nt. and the ending node Ae of the target action paths in the constructed graph (AA, The starting nodes 1 NA, are determined by matching the actions predicted step (Section 4.1) I) and each graph node (action). Similarly, we find the ending node Ae by match- ing each graph node (action) and the action of the future event F detected by the widely used tool, StanfordNLP (Manning al., 2014). Secondly, find all the connected action paths between the starting nodes ) and the ending node Ae with Dijkstra's 's algorithm. Using the starting node As; and the ending node Ae example, the path-finding process is:
because there are
P = softmax (MLP (t))
(2)
where the MLPs ac is the MLP applied specifically for the i-th action prediction. The p'ac is the proba- bility of action existence or not.
4.2 Non- Non-parametric Rule Reasoning
As A. Di ((A,,))))
(5) Finally, we review all the found action paths again for violations of the action chains the action chain set S and delete them. Notably, for the rule paths (like Ap + Ai + Ar and Ag + Aj + A&) involving combinatorial inference rules (like A A, A&) , these action paths need to be merged into one and then checked. After merging, checking, and deleting, multiple action paths may be preserved. They need be further verified the next review module.
The rule-based reasoning of the ART task requires the model not to memorize the rules in the training set and has strong interpretability. Towards this end, we propose the parameerie rule reason- ing network, based on the traditional graph theory rather than the neural network.
In detail, the action set contained in the rule set (action chain set) is represented as A - [A NA As shown in the the Figure 3, the aarpaarmetric rule reasoning network contains three steps: (1) Action Chain Storage. In order facilitate sub- sequent processing, we split the action chains into multiple single-step relational maps and store them in the memory. Taking the action chain Ai - Az... 7 Ar as an example, the splitting
Experiments
We experiment with our ARTNet model on our proposed Video-ART dataset to verify the

---

model's effectiveness for the rule bAsed futuRe- inference deducTiion task (ART). All experi- mental environments are deployed Hikvision
#
Methods
Trans.
Event ACC
Exp
Real Life Socne
VIOLINet (Yang al. 2022)
29.2
22.9
httpss//wwww..ikisiiooocoomm
SCL 2021)
31.3
Dataset. The Video-ART dataset sists of data from real life scenes and movie scenes, which are randomly divided into 11, 02097/706/33 238 (train/val/test) and 3, 902/ 902//34/1 671 (train/val/test), respectively. As stated in Section 3.1, both types of data have their own characteristics. To comprchensively evaluate the performance of the models, we conduct experiments in both scenarios.
3 FAIRR (Sanyal et 20222b
35.8
30.1
4 LF-VILA (Sun
37.9
34,5
5 ARTNet (Ours)
42.3
41.0
Movic Scene
VIOLINet (Yang 2022)
30.1
24.6
2 AHGN_SCC al., 2021)
32.7
26.2
3 FAIRR (Sauyal et al., 2022b)
36.2
32.5
LF-VILA (Sun al.)
37.8
5 ARTNet (Ours)
42.5
42.0
Evaluation Metrics. Following previous de- ductive reasoning tasks (Sanyal et al., 2022b), our ART task requires that the correct future event in the candidate set and its explanation for the rea- soning process both unique. Therefore, "accuracy" to measure the correctness of the model predictions for them. The accuracy of the future event prediction and the explanation prediction are denoted as "Event "Event_ACC" AAC" and "Exp_At "Exp_ACC", respec- tively. We consider an explanation prediction to be potentially correct only when future events are predicted accurately.
Table 2:
Comparison results between ARTNet and the state-of-the-arts. "Trans." indicates the transformer- based architecture.
#
Methods
Trans. EventACC Exp_ACC
Real Life Soene + Movie Scene
VIOLINet (Yang et al., 2022)
30.4
26.7
2 ATIG SCL
2021)
33.5
28.6
3 FAIRR (Sanyal et al., 2022b)
35.7
32.1
4 LF-VILA (Sun et al.)
38.2
35.8
ARTNet (Ours)
42.5
41.8
Baselines. Previous methods from other tasks cannot adopt our ART task a direct manner. Thus, several -tate-of-the-art multi-modal and rea- soning models are extended as the baselines to com- pare. Specifically, make comprehensive com- parison, take into account the following meth- ods: (1) video-language inference methods: LF- VILA (Sun al.), SIGN_SCL (Li et al., 2021), VIOLINet (Liu al., 2020); (2) deductive reason- ing methods: FAIRR (Sanyal et al., 2022b).
Table 3: Comparison results between ARTNet and the state-of-the-arts on the merged dataset the two scenarios. "Trans." indicates the transformer-based chitecture.
the explanation for the reasoning process. (B) The baselines are extended from other tasks and lack targeted domain knowledge of our ART task, which leads to unsatisfactory performance.
Comparison with baselines on the merged dataset. We are interested in the ARTNet model performance the whole dataset. Thus, we merge the two dataset parts, including the real-life ex- amples and the movie examples, and experiment on them. The results of the baseline comparison are shown in Table From the table, we have the following findings: (A) Compared with other baselines, our ARTNet model performs best. This once again demonstrates the rationality of two task- targeted modules, the showledge-guicee Target Perception, and the Non-parametrie Rule Reason- ing, in the ARTNet model. merged dataset of the scenarios does not signif- icantly improve the model performance, compared with being trained on a single scenario. It means the model cannot effectively transfer the learned
5.1 Performance Comparison
Comparison with State-of-the-arrs Our ARTNet model is compared with the baselines on the Video- ART dataset for the ART task. The experiment results are shown in Table From the table, there are the following findings. (A) Compared with the baselines, our ARTNet model performs best and improves the accuracy by more than 4 points on all metrics. We contribute the improvement to 1) the commonsense knowledge guidance, which makes ARTNet focus on the target person and correctly identify the key actions from the video according the future event; 2) the non-paran nonpparametric rule rea- soning, which implements layer-by-layer reasoning strictly according to the given rules and provides
(B) Training on the

---

ARTHet
TARRR
Observed Phenomena (O)
0.5
Event ACC
40.0
Exp ACC.CC
-5
35.0
UR)
325
Rule After cooking, will meal.
100
20
100
Rule 2: Someose will contimue lag another passon.
Percentage
Percentage
Rule 3. Aller cooking, somcoss will have meal up and then de the washing up.
Figure 4: Comparison with the state-of-the-arts on dif-
ferent proportions of training set in the movie scene.
Candidate Futture Events: (Red Text the ecior cruse.)
Event 1: There adult wasten is cost doing the wailing
weating
standing
Event 3: The adult woman in black punt concaniig to bug auother pesson.
There ahut
black doing waehing
Real Life Scene
Movie Scene
CKG
RRN
Corvect Event:
Forre
ARTNNe (Ous)
Event ACC
Exp ECC_ACC
Event_ACC
Exp_ACC
Reasoning Process:
(Observation,
Event
37.7
34,9
38.1
35.1
39.5
37.8
40.5
38.3
Knowiatee Prodictia:
Commonntatoo
wommun, long hair, long sleeve black cast, no glaesos
41.4
40.8
42.3
41.0
42.5
42.0
Figure 5: Case study of the ARTNet performance.
Table 4:
Ablation study of ARTNet on the Video- ART dataset. CKG is the Commonsense- eennolssns--Knowlldge Guidance, and RRN represents the non-parametric Rule Reasoning Network.
5.2 In-depth Analysis
Ablation Study We are interested the contri- bution each key module in our ARTNet model and design the ablation study. Specifically, we surgically remove the Commonsense Heanmonnenns-Knowlldegge Guidance (CKG) and the non-parametric Rule Reasoning Network (RRN) from our ARTNet model and get different architectures. Without RRN, the ARTNet model totally losses the rules exploit capabilities, which is necessary for the ART task. Thus, we replace the RRN module with the advanced NLP model, transformer (Vaswani et al., 2017), rather than simply removing it. The exper- imental results of the ablation study are shown in Table 4. According to the results, there several findings: (A) After removing any key modules, the model performs worse. proves that CKG can improve the model's ability perceive key video information and RRN is able to achieve rigorous reasoning. Thus, both of them are indispensable. (B) When we replace RRN with transformer, the transformer model remembers the rule sets of the examples in the training set during training. This may cause the model to fail to strictly utilize the rule sets given by the test examples during infer- ence, resulting in decrease in accuracy. (C) It more efficient to use key modules together than to use them separately. This demonstrates that the ART task completion can significantly improved by combining CKG's information perception abil- ity with RRN's 's knowledge reasoning ability.
knowledge between the scenarios, which is due the significant differences between the two sce- narios: (1) Movie videos have higher resolution, while real-life vidcos are limited by the lower reso- lution of the shooting devices. This results in the difference in the model's visual perception of the two scenarios.
(2) Movie scenes have richer and more exaggerated plotlines, while real-life scenes have simpler plotlines. Thus, there significant differences in the event reasoning of the two sce- narios.
Comparison with baselines different train- ing data volumes. To evaluate the performance of the ARTNet model trained on different data vol- umes, we randomly select 25%, 50%, and 75% of the training data in the movie scene for experi- ments. The comparison results between ARTNet and baselines are shown in Figure 4. From it, we can observe that the accuracy of ARTNet is still higher than the state-of- state-of-the-arts with the low train- ing data volume. We attribute it to the effectiveness of the knowledge-guided auxiliary training for the transformer module and the non-parametric rule reasoning network independent of the training data, which guarantees the model performance with less training data.
Case Study To further demonstrate the effective-

---

ness of our ARTNet model in visual, we carefully design the case study, Specifically, we select an example from the Video-ART dataset and show the experiment results in Figure 5. From the figure, we can observe that our ARTNet model predictions are completely accurate. This intuitively demon- strates that ARTNet can achieve precise perception and rigorous reasoning for the ART task. In ad- dition, we compare our ARTNet model with the oate--o-thhearr and the ablation base model. These examples are shown in the appendix.
Ruth MJ Byrne 1989. Human deductive reasoning. The Irish Journal of Psychology, 10(2):216-23))
Francesco Calimeri, Francesco Cauteruccio, Luca Cinelli, Aldo Marzullo, Claudio Stamile, Giorgio Terracina, Francoise Sappey-Marinier. 2021. A logic-based framework leveraging neural networks for studying the evolution of neurological disorders. Theory and Practice of Logic Programming. 21-0):80-124
and Dominique
Junwen Chen and Yu Kong. 2021. Explainable video entailment with grounded visual evidence. In Pro- ceedings of the IEEE/CVF International Conference on Computer Vision
6 Conclusion
Yu- Ten Chen, Hsuan- Kung Yang, Chu-Chi Chiu, and Chun-Yi Yi Lec. 2022. S2f2: Single-stage flow forecast- ing for future multiple trajectories prediction. In Pro- ceedings of the IEÉE/CVT Conference on Computer Vision and Pattern Recognition, pages 2536-2533
We study the deductive reasoning process in hu- mans and propose a video-text deductive reasoning task, ART, which is an early exploration of de- ductive reasoning in the field of multi-modal. To promote this new task development, we propose a new dataset, Video-ART, and a strong baseline called ARTNet. Experiments prove the ARTNet effectiveness.
Monireh Ebrahimi, Md Kamnuzzaman Sarker, Federico Bianchi, Ning Xie, Aaron Eberhart, Derek Doran, HyeongSik Kim, and Pascal Hitzler. 2021. Neuro- symbolic deductive reasoning for cross- goos--nooweedee graph entailment. In AAAI Spring Symposium: Com- bining Machine Learning with Knowledge Engineer- ing.
Limitations
We propose a strong baseline, ARTNet, for the ART task, as a field foundation. The ARTNet baseline is limited to approximate the rules as the action chains to further process. In the future, we will update the ARTNet to improve the design of this part. We hope our work could promote the development of the multi-modal deductive reasoning.
Dave Epstein and Carl Vondrick. 2021. Learning goals from failure. In Proceedings of the IEEE/CVF Con- ference on Compueer Vision and Pattern Recognition. pages 11194-11204.
Jonathan St BT Evans, Stephen E Newstead, and Ruth MJ Byrne. 1993. Human reavoning: The psy- chology of deduction. Psychology Press.
Tejas Gokhale, Abhishek Chaudhary, Pratyay Baner- jee, Chitta Baral, and Yezhou Yang. 2022. Semanti- cally distributed robust optimization for vision-and- language inference. In Findings of the Association for Computational Linguisties: ACL 2022, pages 1493-1513.
A Aeknowledgments
Our research is funded in part by the Key Project of the National Natural Science Foundation (No.6203700),, and the Starry Night Science Fund at Shanghai Institute for Advanced Study (Zhejiang University).
Wei Ji, Renjie Liang, Lizi Liao, Hao Fei, and Fuli Feng. 2023a. Partial annotation-basee video moment re- trieval via iterative learning. In Proceedings of the 37th ACM international conference on Multimedia.
References
Wei Ji, Renjie Liang, Zhedong Zheng, Wenqiao Zhang, Shengyu Zhang, Juncheng Li, Mengze Li, and Tat- seng Chua. 2023b. Are binary annotations suf- ficient? video moment retrieval via hicrarchical sicertainty-based active learning. In Proceedings of the IEÉE/CVF Conference on Computer Vision and Pattern Recognition, pages 23013-23022.
Yazan Abu Farha, Alexander Richard, and Juergen Gall. 2018. When will you do anticiinnii temporal occurrences of activities. In Proceedings of the IEEE conference on computer vision and pattern recogni- tion, pages 5343-5352.
Alexandre Alahi, Kratarth Goel, Vignesh Ramanathan, Alexandre Robicquet, Li Fei-Fei, and Silvio Savarese. 2016. Social Istm: Human trajectory prediction in crowded spaces. In Proceedings of the IEEE con- ference on computer vision and pattern recognition, pages 961 961-971.
Wei Ji, Xiangyan Liu, An Zhang, Yinwei Wei, and Xiang Wang. 2023c. Online distillation-enhanneed multi-modal transformer for sequential recommenda- tion. In Proceedings of the 31th ACM international conference on Mullimedia.

---

Wengi Jia, Miao Liu, and James M Rehg. 2022. Genera- tive adversarial network for future hand segmentation from egocentric video. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Purt XIII. pages 639-656. Springer.
Mengze Li, Tianbao Wang, Haoyu Zhang. Shengyu Zhang, Zhou Zhao. Jiaxu Miao, Wenqiao Zhang. Wenming Tan, Jin Wang, Peng Wang, et al. 2022b. End-to-end modeling via information tree for one- shot natural language spatial video grounding. arXiv preprint arXiv: 2xxx:2233000013.
BycoungDo Kim, Scong Hyeon Park, Seokhwan Lee, Elbek Khoshimjonov, Dongsuk Kum, Junsoo Kim, Jeong Soo Kim, and Jun Won Choi. 2021. Lapred: Lane-aware prediction of multi-modal future trajec- tories of dynamic agents. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pat- lern Recognition, pages 14636-14644
Mengze Li, Tianbao Wang. Haoyu Zhang, Shengyu Zhang, Zhou Zhao, Wenqiao Zhang, Jiaxu Miao, Shillang Pu, and Fei Wu. 2022c. Hero: Hierarchi- cal spatio-temporal reasoning with contrastive action correspondence for end-to-end video object ground- ing. In Proceedings of the 30th ACM International Conference on Multimedia, pages 3801-3810.
Kris M. Kitani, Brian Ziehart, James Andrew Bagnell, and Martial Hebert. 2012. Activity forecasting, In Computer Vision ECCV 2012: 12th European Con- ference on Computer Vision, Florence, Italy, October 7-13, 2012, Proceedings, Part IV 12, pages 201-214, Springer.
Jingzhou Liu, Wenhu Chen, Yu Cheng, Zhe Gan, Licheng Yu, Yiming Yang, and Jingjing Liu. 2020. Violin: A large-scale dataset for video-and-language inference. In Proceedings of the IEEF/CVF Confer- ence on Computer Vision and Pattern Recognition, pages 10900-10910.
Tian Lan, Tsung- Loung-Chuan Chen, and Silvio Savarese. 2014. A hierarchical representation for future action prediction. In Computer Vision-ECCV 2014: 13th European Conference, Zurich, Switzerland, Septem- ber 6-12, 2014, Proceedings, Part III 13, pages 689- 704. Springer.
Christopher D Manning, Mihai Surdeanu, John Bauer, Jenny Rose Finkel, Steven Bethard, and David Mc- Closky. 2014. The stanford corenlp natural language processing toolkit. Proceedings of 52nd annual meeting of the association for computational linguis- lics: system demonstrations, pages 55-60.
Juncheng Li, Siliang Tang. Linchao Zhu, Haochen Shi, Xuanwen Huang, Fei Wu, Yi Yang, and Yueting Zhuang. 2021. Adaptive hierarchical graph reason- ing with semantic coherence for video-and -nd--and--dadaa lan-uage inference. In Proceedings of the IEEE/CVF Interna- rional Conference Computer Vision, pages 1867- 1877.
Jiaxu Miao, Xiaohan Wang, Yu Wu, Wei Li, Xu Zhang. Yunchao Wei, and Yi Yang. 2022. Large-scale video panoptic segmentation in the wild: A benchmark. In Proceedings of the IEEE/CVF Conference on Com- puter Vision and Pattern Recognition, pages 21033- 21043.
Lihuan Li, Maurice Pagnucco, and Yang Song. 2022a. Graph-based spatial transformer with memory replay for multi-future pedestrian trajectory prediction. In Proceedings of the IEEE/CVF Conference Com- puter Vision and Pattern Recognition, pages 2231- 2241.
Jiaxu Miao, Yunchao Wei, Yu Wu, Chen Liang, Guan- grui Li, and Yi Yang. 2021. Vspw: A large-scale dataset for video scene parsing in the wild. In Pro- ceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 4133-4143.
Neehar Peri, Jonathon Luiten, Mengtian Li, Aljoša Ošep, Laura Leal-Taixé, and Deva Ramanan. 2022. Forecasting from lidar via future object detection. In Proceedings of the IEEE/CVF Conference on Com- puter Vision and Pattern Recognition, pages 17202- 17211.
Linjie Li, Yen-Chun Chen, Yu Cheng, Zhe Gan, Licheng Yu, and Jingjing Liu. 2020. Hero: Hierarchical en- coder for video+ language onni-representaaiion pre- training. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 2046-2065.
Lance J Rips. 1994. The psychology of proof: Deductive reasoning in human thinking. Mit Press.
Mengze Li, Han Wang, Wenqiao Zhang, Jiaxu Miao, Zhou Zhao, Shengyu Zhang, Wei Ji. and Fei Wu. 2023a. Winner:
hierarchical de- composition and alignment for spatio-temporal video grounding. In Proceedings of the IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition, pages 23090-23099.
Lawrence D Rosenblum. 2011. See what I I'm saying: The extraordinary powers of our five senses. WW Norton & Company.
Michael S Ryoo. 2011. Human activity prediction: Early recognition of ongoing activities from stream- ing videos. In 2011 international conference on com- puter vision. pages 1043. IEEE
Mengze Li, Tianbao Wang, Jiahe Xu, Kairong Han, Shengyu Zhang, Zhou Zhao, Jiaxu Miao, Wenqiao Zhang, Shiliang Pu, and Fei Wu. 2023b. Multi-modal action chain abductive reasoning. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pages 461 4628-4628.
Soumya Sanyal, Zeyi Liao, and Xiang Ren. 2022a. Robustir: Evaluating robustness to logical pertur- bation deductive reasoning. arXiv preprint arXiv:2205.125000

---

Soumya Sanyal, Harman Singh, and Xiang Ren. 2022b. Fairr: Faithful and robust deductive reasoning over natural language. In Proceedings of the 60th Annual Meeting of the Association for Computational Lin- guistics ( (Volume I: Long Papers), pages 1075-1093.
language explanations. In Proceedings of the 30th ACM International Conference on Multimedia, pages 3587-3597.
Wenqiao Zhang, Haochen Shi, Jiannan Guo, Shengyu Zhang, Qingpeng Cai, Juncheng Li, Sihui Luo, and Yueting Zhuang. 2022. Magic: Multimodal rela- tional graph adversarial inference for diverse and unpaired text- rext-based image captioning. In Proceed- ings of the AAAI Conference on Artificial Intelligence, volume 36, pages 3335-3343.
Gunnar A Sigurdsson, Gül Varol, Xiaolong Wang, Ali Farhadi, Ivan Laptev, and Abhinav Gupta. 2016. Hol- lywood in homes: Crowdsoureing data collection for activity understanding. In European Conference on Computer Vision, pages 510-526. Springer.
Yuchong Sun, Hongwei Xue. Ruihua Song, Bei Liu, Huan Yang, and Jianlong Fu. Long-form video- language pre-training with multimodal temporal con- trastive learning. In Advances in Neural Information Processing Systems.
Wenqiao Zhang, Haochen Shi, Siliang Tang. Jun Xiao, Qiang Yu, and Yueting Zhuang. 2021. Consensus graph representation learning for better grounded im- age captioning. In Proceedings of the AAAI Con- ference on Artificial Intelligence, volume 35, pages 3394-3402.
Dídac Surís, Ruoshi Liu, and Carl Vondrick. 2021. Learning the predictability of the future. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12607-12617.
Wenqiao Zhang, Siliang Tang, Yanpeng Cao, Shiliang Pu, Fei Wu, and Yueting Zhuang. 2019. Frame aug- mented alternating attention network for video ques- tion answering. IEEE Transactions on Multimedia, 1234):1032-1041.
Zineng Tang. Jie Lei, and Mohit Bansal. 2021. Decem- bert: Learning from noisy instructional videos via dense captions and entropy minimization. In Pro- ceedings of the 2021 Conference of the North Amer- ican Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2415-2426.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention all you need. Advances in neural information processing systems, 30.
Carl Vondrick, Hamed Pirsiavash, and Antonio Torralba. 2016. Generating videos with scene dynamics. Ad- vances in neural information processing systems, 29.
Yiquan Wu, Kun Kuang, Yating Zhang, Xiaozhong Liu, Changlong Sun, Jun Xiao, Yueting Zhuang, Luo Si, and Fei Wu. 2020. De-biased court's 's view generation with causality. In Proceedings of the 2020 Confer- ence on Empirical Methods in Natural Language Processing (EMNLP), pages 763-780.
Yiquan WU, Yifei Liu, Weiming Lu, Yating Zhang. Jun Feng, Changlong Sun, Fei Wu, and Kun Kuang. 2022. Towards interactivity and interpretability: rationale-based legal judgment prediction framework. In Proceedings of the 2022 Conference on Empiri- cal Methods in Natural Language Processing, pages 4799-
Yiquan Wu, Weiming Lu, Yating Zhang, Adam Jatowt, Jun Feng, Changlong Sun, Fei Wu, and Kun Kuang. 2023. Focus-aware response generation in inquiry conversation. Findings of the Association for Computational Linguistics: ACI. 2023, pages 12585- 12599.
Qian Yang, Yunxin Li, Baotian Hu, Lin Ma, Yuxin Ding, and Min Zhang. 2022. Chunk-aware alignment and lexical constraint for visual entailment with natural


Reviews:
[{"review_id": "dv874sKGMd", "reviewer": "Reviewer_Gibb", "summary": "", "questions": "A. Is it possible to provide the IAA for the data collection process? I saw the details in the appendix to ensure data quality, but it would be great to know a bit more statistically.\n\nB. How did the authors get all rules (i.e. rule storage)? Is it from the whole dataset (i.e. dataset dependent)? \n", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## Rebuttal to Reviewer Gibb\n\nThank you for your patient feedback! We are delighted to offer detailed responses and make comprehensive revisions to our paper based on these valuable suggestions.\n\n## 1. The proposed ART task is close to the well-established video inference task but with more details in commonsense rules and reasoning process. Though it is an interesting direction, based on the restrictions, it might be a bit hard to scale for future researchers.\n\nWe understand your concerns about how to follow our work for future researchers. We will provide a detailed and careful response to the concerns:\n\n***Proposing new models for the ART task.*** We propose a strong baseline, ARTNet, and evaluate the performance of our model and several state-of-the-arts on our Video-ART dataset. In the future, we plan to release the code of our ARTNet and other state-of-the-arts, for the convenience of future researchers. \nAs claimed in our paper, our ARTNet is a strong baseline for the ART task. \nThe current model's accuracy is far from reaching its upper limit. Perhaps there are better methods for rule utilization and video detail analysis that can enhance the model's capabilities. We hope that based on our ARTNet, more researchers will propose higher-accuracy models.\n\n***Proposing new datasets for the ART task*** Compared with the well-established video inference task, our CORE task requires commonsense rule annotations, which brings difficulty to the annotation. However, deductive reasoning is a basic recognition ability of humans, detailed in lines 55-57 of our paper. In practice, many annotators with certain reasoning abilities can annotate the commonsense rule labels and other deductive reasoning annotations effectively. With a sufficient number of such annotators or ample time, the deductive reasoning dataset can be substantially expanded. In the future, if other researchers would like to expand our dataset, we will provide enough help with training annotators, quality control checks, and other related measures.\n\n***Contribution to other related tasks*** We annotate various types of commonsense knowledge, including sentiment and actions. This allows our work to provide a solid data foundation for tasks such as video emotion recognition or action recognition.\n\n## 2. Is it possible to provide the IAA for the data collection process? I saw the details in the appendix to ensure data quality, but it would be great to know a bit more statistically.\n\nYes, we are happy to introduce the annotation process in detail. We guarantee the dataset quality at each stage of its construction:\n\n***Data Collection***\n1. ***Personnel Training.*** \nWe provide separate centralized training for data collectors and verifiers, informing them of the standards for data collection and verification. Specifically, there are many collection standards to follow. For example, the collected videos need to be sufficiently clear and information-rich to avoid prolonged periods of low-information static scenarios. For the verification standards, the verifiers need to check each collected video one by one to ensure the quality, based on the collection standards. \n\n2. ***Double Verifications.***\nThere are double verifications for the collected videos. Specifically, except for the verification of the verifiers, the trainers are responsible for conducting random spot checks on 30% of data that has already undergone comprehensive verification. We hope the double verification can ensure the quality of the collected videos.\n\n***Data Annotation***\n1. ***Multiple Training Sessions.*** \nThe annotators are trained for several times. Before annotation, we provide centralized training for all annotators. After training, we conduct annotation tests for them and select annotators with high accuracy for the subsequent annotation. During the annotation process, if new issues are identified by verifiers, we will conduct centralized retraining for the annotators and perform comprehensive revisions for annotated examples. For example, different annotators have different standards for \"blouse\" and \"shirt\". The trainers find reference images as standards to retrain the annotators, and require them to perform rework. \n\n2. ***Double Verifications.*** \nSimilar to the process of data collection, there are double verifications in the annotation process. Firstly, the verifiers check the annotated examples one by one. Then, the trainers of the verifiers randomly sample 30% of the labeled examples, and check them carefully to further verify the annotation quality. \n\nWith rigorous personnel training, we found that the agreement rate between annotators and verifiers reached 95% for the same annotation test. We will add these annotation details to our paper to completely introduce our dataset. Thank you for the valuable suggestions!\n\n\n## 3. How did the authors get all rules (i.e. rule storage)? Is it from the whole dataset (i.e. dataset dependent)?\n\n### a) How did the authors get all rules (i.e. rule storage)?\n\nWe construct the rule storage based on the rule set of the input example.\nSpecifically, following the previous deductive reasoning task [1] in the NLP field, each example is accompanied by an independent rule set. We transform this rule set into many action chains, detailed in lines 303-319 of our paper. When analyzing each input example, the rule storage is constructed based on the action chains corresponding to the given example. \n\nWe are sorry for the unclear description of the rule storage construction. We will add the detailed construction process into the main body of our paper.\n\n### b) Is it from the whole dataset (i.e. dataset dependent)?\n\nNo, the rule storage is constructed based on the rule set of the input example, detailed in question 3-a). \n\n## 4. L207: Would like to know a bit of copyright related discussions for the 3,983 manually collected video clips (i.e. are those 80 American movies free to use and distribute without license issues?)\n\nAbout the copyright of our dataset, we plan to follow the protection method of the widely known dataset, ImageNet [2], whose examples are collected from the internet. Specifically, we provide the thumbnail frames of the videos and a copyright infringement takedown notice. Only after signing an agreement and guaranteeing that they will not commercially use our dataset, we will provide the original video links, as well as the start and end timestamps of the used video clips. \n\n## 5. L243, L256: It is minor, but for more broader audiences, wording like from the top 50 universities in the world sounds a bit not appropriate.\n\nThank you for the valuable reminder! We are sorry for the inappropriate description. We will change it to \"the undergraduate and graduate students with strong reasoning abilities\" in the next paper version. Thank you!\n\n[1] Sanyal S, Singh H, Ren X. Fairr: Faithful and robust deductive reasoning over natural language[J]. arXiv preprint arXiv:2203.10261, 2022.\n\n[2] Deng J, Dong W, Socher R, et al. Imagenet: A large-scale hierarchical image database[C]//2009 IEEE conference on computer vision and pattern recognition. Ieee, 2009: 248-255.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper formulates the ART task as an early exploration of deductive reasoning in the multi-modal domain. This work provides a large and rich annotated dataset which could benefit the research related to this topic. Also, authors develop a well-designed baseline towards better understanding the task. Overall it is complete and solid from my point of view.", "reasons_to_reject": "The proposed ART task is close to the well-established video inference task but with more details in commonsense rules and reasoning process. Though it is an interesting direction, based on the restrictions, it might be a bit hard to scale for future researchers.", "questions_to_authors": "A. Is it possible to provide the IAA for the data collection process? I saw the details in the appendix to ensure data quality, but it would be great to know a bit more statistically.\n\nB. How did the authors get all rules (i.e. rule storage)? Is it from the whole dataset (i.e. dataset dependent)? \n", "paper_topic_and_main_contributions": "This work proposes a new ART task (rule based future-inference deduction task) under the multi-modal setting. It collects a dataset called Video-ART consisting of 23,895 examples with rule set and reasoning process annotations. Based on it, a strong baseline model ARTNet is also developed with three modules: 1) knowledge-guided target perception with encoder-decoder transformers, 2) rule-based graph reasoning network, and 3) reasoning path review to decide the correct future event.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "A. L207: Would like to know a bit of copyright related discussions for the 3,983 manually collected video clips (i.e. are those 80 American movies free to use and distribute without license issues?)\n\nB. L243, L256: It is minor, but for more broader audiences, wording like `from the top 50 universities in the world` sounds a bit not appropriate. \n"}}, {"review_id": "tbDgVB4auy", "reviewer": "Reviewer_XipY", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## Rebuttal to Reviewer XipY\n\nWe appreciate your valuable suggestions. Thank you very much! We will carefully respond to the comments item by item. \n\n## 1. The setting of confusing items in the candidate events is a critical aspect of the dataset quality. However, the paper lacks a detailed description of this aspect. The provided cases in the paper seem to contain relatively unconfusing incorrect candidates, that is, it seems easy to distinguish the correct candidate from the incorrect ones based on relevance or similarity to the rules. Addressing this concern would strengthen the paper's dataset construction.\n\n### a) The setting of confusing items in the candidate events is a critical aspect of the dataset quality. However, the paper lacks a detailed description of this aspect.\n\nWe understand your concerns and are sorry for the incomplete description of the confusing item construction. We will describe the construction process in detail.\n\nBefore generating a confusing item, a correct event is constructed. Then, we inject an error into the correct event.\nThere are several ways of injecting errors to choose:\n\n1. Change the correct appearance and clothing description into the error one, like the description of another character in the video. \n\n2. Replace the correct rule-based reasoning process with a logically inconsistent rule-based reasoning process. Then, change the correct event into the confusing event corresponding to the logically inconsistent reasoning process.\n\n3. Replace the correct rule-based reasoning process with another logically consistent reasoning process that contradicts the video details. Then, change the correct event into the confusing event corresponding to the new reasoning process. \n\n4. Inject some description contradicting the video details into the correct item and change it into a confusing item.\n\n5. Inject some description contradicted with the common sense into the correct item and change it into a confusing item. \n\n6. We also encourage the annotators to choose other error injection methods specific to the particular videos, such as contradicting the scenes in the videos.\n\nThe verifiers check all the confusing items carefully, to ensure the quality of these items.\n\n\n### b) The provided cases in the paper seem to contain relatively unconfusing incorrect candidates, that is, it seems easy to distinguish the correct candidate from the incorrect ones based on relevance or similarity to the rules. Addressing this concern would strengthen the paper's dataset construction.\n\nWe understand your concerns about that there are some unconfusing items in our cases. We randomly select the cases from our dataset to present our model performance. \nThese individual cases cannot effectively represent our entire dataset from the perspective of the candidate event difficulty. \nOur dataset includes examples and candidate events of various difficulty levels. Many candidate events are difficult to distinguish, which requires a combination of selecting appropriate rules and considering video details to make accurate inferences. \nWe apologize for the incomplete selection of visual cases. \nWe will add more cases to the main body of the next paper version, to completely show examples of different difficulty levels.\n\nIn addition, to provide the testbed for both the early-stage models (poor performance) and later-stage models (good performance), there are both easy-to-judge and difficult-to-judge candidate events in our dataset. From the perspective of model accuracy, the current dataset has a moderate level of difficulty. It distinguishes the inference capabilities of the current-stage models while leaving room for future research advancements.\nWhen the models reach the accuracy limit on our dataset, we plan to construct a more challenging dataset to further promote the development of this field.\n\n\n## 2. The paper lacks analyses on dataset bias. First, it remains unclear how crucial each modality (image or text) is to the new task. Evaluating the performance of models that solely rely on image or text inputs would provide insights into the importance of each modality. Second, there is a potential issue of models exploiting the relevance or similarity between candidates and rules as a shortcut, rather than genuinely reasoning through the problem. Evaluating and discussing this dataset bias is essential to ensure the dataset quality.\n\n\n\n### a) The paper lacks analyses on dataset bias. First, it remains unclear how crucial each modality (image or text) is to the new task. Evaluating the performance of models that solely rely on image or text inputs would provide insights into the importance of each modality. \n\nThank you for the reminder! We will complete the analysis of each modality function in the model reasoning process:\n\n***Analysis of the function of the input visual information.*** We black out all the videos in our dataset. Then, we train and evaluate our model ARTNet with the text and these masked videos, to verify the function of the input videos. The experiment results in the real-life scene are shown in the following:\n\n| Methods | Event_ACC | Exp_ACC |\n| ------ | ------ | ------ |\n| ARTNet | 42.3 | 41.0 |\n| ARTNet (with masked videos) | 27.3 | 24.1 |\n\nVisual information serves as the foundation for our task's reasoning process. After blacking out the input videos, the accuracy of our ARTNet model has significantly declined. This proves the importance of visual information in the inference process of the AI system.\n\n***Analysis of the function of the input text information.*** The ART task requires the AI system to judge the correctness of all candidate events and select the correct one. Thus, it is necessary for the AI system to get the input of the candidate event text. Thus, when verifying the effectiveness of the input text, we only remove the participation of the textual rule set, during the training and testing process of our ARTNet. The experiment results in the real-life scene are shown in the following:\n\n| Methods | Event_ACC | Exp_ACC |\n| ------ | ------ | ------ |\n| ARTNet | 42.3 | 41.0 |\n| ARTNet (without rule sets) | 36.2 | - |\n\nIt is worth noting that without the rule sets, the rule-based explanations cannot be inferred by our ARTNet, whose accuracy (Exp_ACC) cannot be calculated.\n\nFrom the experiment results, it can be observed that the ARTNet performance drops significantly without rule sets. It proves that the importance of the textual rule set in the reasoning process of the AI system. The complete experiment results will be added to the next version of our paper. \n\n\n### b) Second, there is a potential issue of models exploiting the relevance or similarity between candidates and rules as a shortcut, rather than genuinely reasoning through the problem. Evaluating and discussing this dataset bias is essential to ensure the dataset quality.\n\nWe understand your concerns about the shortcut of only applying the candidates and rules in the reasoning process. It is necessary to prove the importance of visual information in the reasoning process. As shown in the response to question 2-a), the performance of our ARTNet drops significantly, without the visual information. This is because crucial information for determining the correctness of candidate events, such as the appearance and clothing of target characters, initial actions of characters, etc., must be obtained from the videos. The absence of visual information will result in an incomplete reasoning basis for the AI system.\nWe will add more experiment results and analysis about the genuine reasoning of models to the next version of our paper.\n\n\n## 3. It would be better to incorporate potential applications or downstream tasks of the new task, which may enhance the significance of the new task.\n\nWe appreciate the valuable suggestions! There are numerous potential applications for our work. One significant application is in legal judgment. Specifically, in the judicial scenario, we can predefine rule sets based on various legal provisions, such as environmental protection laws, civil laws, and more. In specific cases, an AI system with deductive reasoning abilities can effectively combine video evidence with rule sets to facilitate the judicial process.\n\nIn the next version of our paper, we will provide a more detailed description of the applications of our ART task in the main body of the paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The proposed task is interesting and has the potential to promote rule-based reasoning in the multimodal domain.\n- The paper is well-written, offering a comprehensive task definition and clear formulation.", "reasons_to_reject": "- The setting of confusing items in the candidate events is a critical aspect of the dataset quality. However, the paper lacks a detailed description of this aspect. The provided cases in the paper seem to contain relatively unconfusing incorrect candidates, that is, it seems easy to distinguish the correct candidate from the incorrect ones based on relevance or similarity to the rules. Addressing this concern would strengthen the paper's dataset construction.\n- The paper lacks analyses on dataset bias. First, it remains unclear how crucial each modality (image or text) is to the new task. Evaluating the performance of models that solely rely on image or text inputs would provide insights into the importance of each modality. Second, there is a potential issue of models exploiting the relevance or similarity between candidates and rules as a shortcut, rather than genuinely reasoning through the problem. Evaluating and discussing this dataset bias is essential to ensure the dataset quality.\n- It would be better to incorporate potential applications or downstream tasks of the new task, which may enhance the significance of the new task.", "questions_to_authors": "", "paper_topic_and_main_contributions": "The paper proposes a new task that deduces future events based on rules and visual information. It constructs a dataset and a baseline model for the new task. Experimental results demonstrate the effectiveness of the proposed model.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "uHGQ9XXceq", "reviewer": "Reviewer_urk3", "summary": "", "questions": "The over claim towards ``the first'' is not a big deal.\nMy main concerns is the real-world application, can you provide a more applicable usage towards your model and task. For example, whether the rule set can be generated by LLMs? And how is the performance of your method when facing generated rule set? Moreover, whether your dataset can be further expanded by LLMs in the aforementioned approach?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## Rebuttal to Reviewer urk3\n\nThank you for your appreciation and patient feedback on this research paper. We will carefully respond to each of the comments one by one.\n\n## 1. The task proposed in this paper cannot be applied in real application, since there are no extra annotation, like the rule set.\n\nWe understand your concerns regarding the difficulty of obtaining the rule set in the real applications. We will clarify how to obtain the rule set and add a more complete and detailed description to the next paper version.\n\nThe rule set in the real-life application can be viewed as a knowledge base. In real-world scenarios, there are several situations involved in obtaining the rule set:\n\n***The rule set is already available.***\nIn many fields, like the judicial/legal scenario, the knowledge base has been well-defined by the experts. After simple filtering and modifications, the rule sets can be built using the existing knowledge base. With the constructed rule sets, the deductive reasoning AI system may assist in completing prediction tasks in the given scenarios, such as judicial rulings.\n\n***The rule set will be defined by humans.***\nIn many critical fields (like counter-terrorism), human needs to participate in the construction of foundational rules. Specifically, in the field of security, humans are required to define different rule sets for various real-world scenes. The AI system can provide crime alerts based on video footage and the rule set specific to this scene. \n\nFurthermore, using LLMs is a good choice to reduce the manual labor cost of annotating rule sets in these fields. Specifically, the well-trained LLMs provide the base rule sets. The human annotators need to make simple corrections or adjustments to the generated rule sets.\n\n## 2. This work is not the first to explore deductive reasoning in the multi-modal domain, please refer to compositional reasoning in VideoQA and ImageQA for more information. These task are not same as your task, but they explore programs to model the deductive reasoning in the multi-modal domain.\n\nWe appreciate your suggestions about comparing with these well-established tasks (VideoQA and ImageQA). We will clarify the difference between these tasks and our ART ask. Then, we will update the complete task comparison to the next paper version.\n\nReferring to the format of the deductive reasoning task in the NLP field [1], we design the multi-modal deductive reasoning, ART. Both our task and the NLP deductive reasoning task [1] require the AI system to perform deductive reasoning under specific explicit rule constraints\n\nBased on this requirement, our ART task has three characteristics different from the VideoQA/ImageQA: (1) the explicit rule set is provided; (2) the rules used for each example are explicitly restricted; (3) the rule-based reasoning process needs to be provided by the AI system. \n\n\n\n## 3. The over claim towards ``the first'' is not a big deal. My main concerns is the real-world application, can you provide a more applicable usage towards your model and task. For example, whether the rule set can be generated by LLMs? And how is the performance of your method when facing generated rule set? Moreover, whether your dataset can be further expanded by LLMs in the aforementioned approach?\n\n### a) The over claim towards ``the first''.\n\nThank you very much for your understanding and reminder! In the next paper version, we will modify the claim about our ART task in the multi-modal deductive reasoning field. Once again, thank you. \n\n\n### b) My main concerns is the real-world application, can you provide a more applicable usage towards your model and task. \n\nWe understand your concerns about the real-life application of our ART task. As described in the response to question 1, one potential application of our work is to adjudicate different types of cases in the judicial field, based, based on different legal provisions. Specifically, according to different legal provisions, the rule sets are defined. The AI system with deductive reasoning abilities can assist judicial decisions based on video evidence and rule sets. We will update our paper and add the real-world application description to the main body. \n\n### c) For example, whether the rule set can be generated by LLMs? And how is the performance of your method when facing generated rule set? \n\nGenerating the rule set using LLMs is an interesting idea, which is the next step of our research. Recently, we plan to contact the Microsoft company and use the GPT-4 to generate more rule sets. We plan to add the complete experiment results on the generated rule sets to the next version of our paper, and discuss the significance of the multimodal LLM in obtaining comprehensive rule sets. \n\nIn real-world applications, the rule sets in certain fields are not suitable to be generated solely using LLMs. For example, in the field of judiciary, rules generated directly by LLMs often fail to align with actual legal provisions. Therefore, judicial rules must be formulated based on legal provisions and undergo meticulous manual review.\n\n### d) Moreover, whether your dataset can be further expanded by LLMs in the aforementioned approach?\n\nThis is also an interesting idea. In the future, we plan to generate more examples based on the GPT-4.\nThe extension includes two aspects: (1) Modifying existing examples in our dataset to generate new ones; (2) Creating entirely new examples based on the generation capabilities of GPT-4.\n\nHowever, even with GPT-4, the AI system may not generate a 100% correct rule set. If so, we also need human annotators and verifiers to further annotate and verify the examples generated by the LLMs. \n\n\n[1] Sanyal S, Singh H, Ren X. Fairr: Faithful and robust deductive reasoning over natural language[J]. arXiv preprint arXiv:2203.10261, 2022.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. This work propose a different deduction reasoning in multi-modal domain.\n2. This work contribute a meaningful dataset to support this task.\n3. Experimental results shows that the deduction reasoning can be finished in this way.", "reasons_to_reject": "1. The task proposed in this paper cannot be applied in real application, since there are no extra annotation, like the rule set.\n2. This work is not the first to explore deductive reasoning in the multi-modal domain, please refer to compositional reasoning in VideoQA and ImageQA for more information. These task are not same as your task, but they explore programs to model the deductive reasoning in the multi-modal domain.", "questions_to_authors": "The over claim towards ``the first'' is not a big deal.\nMy main concerns is the real-world application, can you provide a more applicable usage towards your model and task. For example, whether the rule set can be generated by LLMs? And how is the performance of your method when facing generated rule set? Moreover, whether your dataset can be further expanded by LLMs in the aforementioned approach?", "paper_topic_and_main_contributions": "This paper proposed a  rule based future-inference deduction task to explore the deductive reasoning in multi-modal domain.\nMoreover, this paper construct a large-scale dataset to promote multi-modal deductive reasoning research, which consists of 23, 895 examples where dense annotations including the rule set, reasoning processes, and auxiliary commonsense knowledge are provided.\nFinally, the author contribute a strong baseline, ARTNet, tailored for the ART task.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a new ART task (rule based future-inference deduction task) to explore the deductive reasoning in multi-modal domain. Moreover, this paper construct a large-scale dataset to promote multi-modal deductive reasoning research, which consists of 23, 895 examples where dense annotations including the rule set, reasoning processes, and auxiliary commonsense knowledge are provided. Based on it, a strong baseline model ARTNet is also developed with three modules: 1) knowledge-guided target perception with encoder-decoder transformers, 2) rule-based graph reasoning network, and 3) reasoning path review to decide the correct future event. Experimental results demonstrate the effectiveness of the proposed model.
All reviewers rate soundness and excitement highly. In summary, the paper formulates the ART task as an early exploration of deductive reasoning in the multi-modal domain. This work provides a large and rich annotated dataset which could benefit the research related to this topic. Also, authors develop a well-designed baseline towards better understanding the task. The proposed task is interesting and has the potential to promote rule-based reasoning in the multimodal domain. The paper is well-written, offering a comprehensive task definition and clear formulation. This paper has great potential for accept to main conference.
Decision: Accept-Main