Multi-modal Action Chain Abductive Reasoning
Year: 2023

Abstract:
Abductive Reasoning, has long been considered to be at the core ability of humans, which enables us to infer the most plausible explanation of incomplete known phenomena in daily life. However, such critical reasoning capability is rarely investigated for contemporary AI systems under such limited observations. To facilitate this research community, this paper sheds new light on Abductive Reasoning by studying a new vision-language task, Multi-modal Action chain abductive Reasoning (MAR), together with a large-scale Abductive Reasoning dataset: Given an incomplete set of language described events, MAR aims to imagine the most plausible event by spatio-temporal grounding in past video and then infer the hypothesis of subsequent action chain that can best explain the language premise. To solve this task, we propose a strong baseline model that realizes MAR from two perspectives: (i) we first introduce the transformer, which learns to encode the observation to imagine the plausible event with explicitly interpretable event grounding in the video based on the commonsense knowledge recognition ability. (ii) To complete the assumption of a follow-up action chain, we design a novel symbolic module that can complete strict derivation of the progressive action chain layer by layer. We conducted extensive experiments on the proposed dataset, and the experimental study shows that the proposed model significantly outperforms existing video-language models in terms of effectiveness on our newly created MAR dataset. Our dataset is available 1. First author. mengzeli@zju.edu.cn ∗Corresponding author. zhaozhou@zju.edu.cn jiaxu.miao@yahoo wenqiaozhang@zju.edu.cn https://github.com/BirdFly16/TO-MAR Observation: Action Chain: The man in a T-shirt choked on food is vomiting into the toilet. chock put down stand up run out run in bend over Event Grounding: Abductive Reasoning Timeline Explanation: (devour the food) t0

heading: 1 Introduction
text: Abductive Reasoning typically begins with an incomplete observation or several observations and then proceeds to the likeliest possible explanation for the set (Du et al., 2021; Peirce, 1974). Given an event observation (O), humans can find some related information in the recollection and easily trace the complete process with strong reasoning ability as a hypothesis (H) to explain the observation. For instance, when we observe the O: "The man in a T-shirt chocked on food is vomiting into the toilet" and remember that he used to devour food in the kitchen, we could infer the complete event chain about that man as hypothesis H: the man devoured the food in the kitchen → he was choking → he put down the food → he left the kitchen in a hurry → he ran into the bathroom → he bent over the toilet. This ability enables us to perform better than machines in high-level reasoning and would be the most precious capacity for modern AI. Therefore, it is important to enhance such Abductive Reasoning capacities of AI models, i.e., complete process as explanation.
Motivated by the aforementioned Abductive Reasoning scenario, we present a novel visionlanguage task, called Multi-modal Action chain abductive Reasoning (MAR), which is illustrated in Figure 1. Specifically, given a set of languagedescribed observations, the MAR task targets to precisely localize the target event in the past video
4617
(visual recollection simulation of human) about the language-described person and rigorously reason out the subsequent action chain (subsequent events inference), to explain the observation. Different from the previous Abductive Reasoning task (Bhagavatula et al., 2019) focusing on the unimodal and partial reasoning, our new task has the following characteristics: (i) MAR needs to locate the target event from the complex video information to explain the textual observation; (ii) MAR requires rigorous recovery of the complete action chain.
These characteristics introduce two challenges to the MAR task: (1) Heterogeneous Information Alignment. To realize the event grounding, aligning the cross-modal information is necessary. However, unlike the highly concise language description, the videos in real scenes usually contain complex and redundant information, including multiple people with different appearances, actions, scenes, etc. Only a small amount of information in videos aligns with the text-described observation. Precisely extracting information from the complicated video information to align is difficult but necessary for the AI system. (2) Action Chain Reasoning. Rigorous action chain reasoning is an interlocking and progressive process. If one step of reasoning is wrong, the correctness of subsequent steps cannot be guaranteed. Therefore, for action chain reasoning, it is highly required to correctly learn the logical relationship between actions and correctly select from multiple next-step actions in each step of reasoning.
We contribute a carefully annotated large-scale dataset, TO-MAR, based on our collected data to facilitate the challenges solved for the MAR task. It contains 14, 201 cross-modal examples based on the videos manually collected from the TV show and the existing dataset (Sigurdsson et al., 2016). To address the MAR task challenges, these examples have targeted manual annotations: (i) Commonsense Knowledge Annotation for Assisted Alignment. We provide the full annotation of commonsense knowledge for every textual observation related person in the large-scale videos, including the character’s appearance, clothing, actions, sentiment, etc. (ii) Rigorous Annotation for Action Chains. Expert annotators with strong logical ability are asked to annotate the language-described observations and the action chains, ensuring the accuracy and rigor of logical annotations.
Based on the constructed dataset, we propose an
end-to-end Neural-symbOlic model Via commonsense knowledgE for multi-modaL action chain Abductive Reasoning (NOVEL). There are two key targeted designs: (i) Knowledge-guided Alignment. We adopt the multi-task learning paradigm to synchronize the recognition learning of commonsense knowledge. Based on such knowledge recognition ability, our NOVEL can minimize the interference of the inferred event and past video, thereby more easily learning to generate explicit event grounding conditioned on textual observations. (ii) Graph-aware Symbolic Reasoning. Motivated by the powerful reasoning ability of the symbolic network (Yi et al., 2018), we design the targeted symbolic reasoning module based on the traditional graph theory. Specifically, we store the learned action association graph in the training process. During inference, we determine the intermediate action chain between the textual observation and the grounded video event with Dijkstra’s algorithm (Dijkstra, 1959). Our contributions are three-fold:
• We introduce a new task, Multi-modal Action chain abductive Reasoning (MAR), which includes two sub-parts: target event grounding and sequential action chain reasoning.
• A carefully collected large-scale dataset, TOMAR, is provided, in which the complete observation-explanation pairs are accurately and rigorously annotated. In addition, a variety of commonsense knowledge that can aid in training is annotated in detail.
• An end-to-end neural-symbolic model named NOVEL is proposed for MAR with knowledge-guided alignment and graph-based symbolic reasoning. Extensive experiments demonstrate the model design rationality.

heading: 2 Related Work
text: Multi-modal Spatio-temporal Grounding. Our MAR task is related to the multi-modal spatiotemporal grounding task, which aims to detect target visual information described by the sentence from the video. It is an important task in the visual understanding domain (Miao et al., 2023, 2021; Zhang et al., 2019, 2022b,a). For video grounding research direction, most researchers focus their research on temporal grounding task (Yang et al., 2021; Xiao et al., 2021). However, spatio-temporal
grounding and spatial grounding (Li et al., 2022a; Yang et al., 2022; Jin et al.; Su et al., 2021; Li et al., 2022b, 2023) have received less attention. (Zhang et al., 2020) uses the graph neural network to model the spatio-temporal relationship between objects to align text descriptions for object localization. In addition, to evaluating the model performance, this paper proposes a complete large-scale dataset. (Su et al., 2021) designs an end-to-end multi-modal grounding model based on the transformer. It outperforms all previous models without pre-training. Later, (Yang et al., 2022) makes a targeted design to fit the pre-trained parameters and achieves a great improvement in accuracy.
Neural-symbolic Reasoning. Compared with the pure neural network (Li et al., 2020b; Wu et al., 2022; Li et al., 2020a; Wu et al., 2020; Miao et al., 2022), neural-symbolic models have stronger inference and perception capabilities. (Yi et al., 2018) is an earlier paper exploring this direction. It stitches symbolic models behind the multi-modal neural network to reason on the information the network perceives. In this neat way, the model achieves excellent results. (Li et al., 2020c) combines the laws of physics with deep learning to make models capable of fitting complex physical processes. In this way, the model can effectively predict the motion trends of objects in the physical world. Similarly, (Ding et al., 2021) uses physical laws such as collision to design a symbolic model to process the information perceived by the neural network, which can effectively predict the future motion of objects such as balls or sliders. (Greff et al., 2019) applies neural network and symbolic model to highlevel and low-level visual relation detection, respectively, and achieves good performance through the cooperation of the two.
Abductive Reasoning. There is a limited amount of existing research work on abductive reasoning AI systems (Du et al., 2021). Previous abductive reasoning tasks (Bhagavatula et al., 2019; Liang et al., 2022) require AI systems to provide a unimodal and partial explanation, which may lack some key information.

heading: 3 Dataset Description
text: To advance research on Abductive Reasoning, we propose the Multi-modal Action chain abductive Reasoning task (MAR) and contribute a large-scale Text-videO dataset for the MAR task (TO-MAR). To complete the MAR task, the AI system needs to
reason out the complete process (the video event and the subsequent action chain) to explain the observed event described by the textual observation O. In detail, the target event Et is grounded in the video V by localizing the temporal boundary T and the target person bounding boxes B in the event Et. After that, the action chain A = {Ai}NAi=1 following the target event Et is inferred, where the NA is the number of actions in chain A.

heading: 3.1 Dataset Preparation
text: We collect and annotate the proposed TO-MAR dataset based on the above MAR’s definition. To increase the variety of data, a two-source dataset collection is conducted for labeling: (1) We select lifestyle videos from the Charades dataset (Sigurdsson et al., 2016). These examples contain diverse people and rich activities. (2) The TV show videos are selected from 92 well-known American dramas, such as The Big Bang Theory, Grey’s Anatomy, etc. Notably, the number of people is relatively limited compared to the first source, but the causal links among events are clearer.

heading: 3.2 Dataset Annotation
text: The TO-MAR dataset contains commonsense knowledge annotations and MAR task annotations. More details are in the appendix.
Commonsense Knowledge Annotation. We annotate the commonsense knowledge that is related to the person conditioned on the observations for event grounding, including the appearance and clothing of key video characters. We also annotated the characters’ actions, sentiments and located scenes in each frame. Each category contains several critical subcategories (e.g., Appearance: Gender, Hair Length, Age. ) recognized
by labeling experts to ensure the MAR model the relationship between the event and observations.
MAR Task Annotation. The MAR task annotation for the video V consists of a languagedescribed observation O and its explanation, including the spatio-temporal markers (the bounding boxes B and the temporal boundary T ) of the target video event Et and the subsequent action chain A. In order to ensure the quality of the annotations, we employ experts with related annotation experience from leading AI research institutions. The annotation process contains two steps: Step 1: Annotating. The annotators annotate the natural language description O of the observation referred to the commonsense knowledge and the follow-up video content for the TV show video clips. The corresponding target video event Et and the subsequent action chain A annotations are also recorded. Step 2: Verification. The validators carefully validate the annotated examples. If the annotation is not agreed upon by the validators, it is relabeled or dropped.
Dataset Features and Statistics. To further introduce the TO-MAR dataset, we analyze the data distribution: (1) Dataset Split. We separate the TO-MAR dataset into train/val/test sets with 12, 527/426/1, 248 labeled examples. The videos of the three sets do not overlap. More detailed statistics are shown in Figure 2. (2) Diversity. To include various causal relationships, our dataset contains rich activities (cooking, work, etc.) and various scenes (family, hospital, etc.). (3) LargeScale. The TO-MAR dataset consists of 14, 201 examples, which proves a testbed for the evaluation of the MAR task models.

heading: 4 Method
text: MAR Task Formulation. Given the language description of the observed event O and the video V consisting of NE past events E = {Ei}NEi=1, the
Multi-modal Action chain abductive Reasoning task (MAR) aims to explain the observation O by localizing the target event (the t-th event) Et in the video V , and supplementing the intermediate action chain A = {Ai}NAi=1 between the target event Et and observation O. All actions in the action chain A are chosen from the action set S = {Si}NSi=1. Meanwhile, the NA and the NS are the action number in the action chain A and the action set S, respectively. To formulate the model training process, we define M as the trained model initialized with parameter Θ. Then, the training optimization function can be expressed as:
M((O,V),(E ,S); Θ) = max
Θ ϵ(ξ(E ,S), δ(O,V; Θ)). (1)
In it, the function ξ(.) outputs the ground truth, which contains: (1) the temporal boundary T of the target video event Et and the target person bounding boxes B in the event Et; (2) the category of each action in the action chain A. The δ(.) outputs the prediction, and the Θ is the learnable parameter. The function ϵ(.) calculates the consistency of ξ(.) and δ(.).
Model Pipeline. As shown in the Figure 3, after extracting features from the observation O and the frames of the video V with RoBERTa (Liu et al., 2019) and Resnet101 (He et al., 2016), our NOVEL M mainly contains two parts to process the features. (1) First, the multi-modal features is reasoned by the transformer model (Vaswani et al., 2017). Based on the recognition ability of the commonsense knowledge (e.g., the character’s actions, appearance, etc.), the model focuses on the key video information aligned with the textual observation O, and learns to infer the temporal boundary T and the target bounding boxes B of the target event Et in the video. (2) Second, the symbolic reasoning part maintains a relation memory module and stores the learned action relation in it at the training step. In the inference phase, the action graph is constructed based on the action relations. We use Dijkstra’s algorithm to find the connected path on the action graph so as to infer the action chain A between the observation O and the target video event Et.

heading: 4.1 Knowledge-guided Alignment
text: We follow the general training and prediction protocol of cross-modal transformer applied in other video grounding methods (Kamath et al., 2021;
Yang et al., 2022). The transformer decoder generates the features for all video frames. For each frame, we predict the bounding boxes and whether it is the temporal start or end of the target event Et.
However, there is complex information in the input video V . We need to guide the model to focus on the video information aligned with the observation O during the training process, using the prediction learning for the commonsense knowledge (e.g., human action, appearance, etc.) of the target character in the video. Specifically, following previous transformer-based models (Yang et al., 2022; Carion et al., 2020), several query vectors are defined: frame level vectors Qr = {qir}Nri=1 and video level vectors Qv = {qiv}2i=1. The Nr is the frame number in the video V . With these query vectors, we apply transformer decoder D to analyze multi-modal features F fused by the transformer encoder:
[Fr,Fv] = D([Qr,Qv],F), (2)
where [.] represents the feature concatenated.
Next, we predict the commonsense knowledge of the language-described character using these output frame level features Fr = {fir}Nri=1 and video level features Fv = {fiv}2i=1. The scene where the character is located and the character’s sentiment change over time. Thus, we predict them frame by frame using MultiLayer Perceptron (MLP). Viewing the i-th frame as an example, the prediction process is represented as:
pisc = softmax(MLPsc(f i r)), (3)
pise = softmax(MLPse(f i r)). (4)
In them, the MLPsc and MLPse are the MLP applied to predict the probabilities of all scene and sentiment classes (pisc and pise). In addition, we apply the video level features Fv to predict the appearance and clothing of the target character described by the language sentence O, and the character’s action in the target video event Et:
pap = softmax(MLPap(f 1 v)) (5)
pcl = softmax(MLPcl(f 1 v)), (6)
pac = softmax(MLPac(f 2 v)). (7)
In them, the MLPap, MLPcl and MLPac are the MLP applied to predict the probabilities of all appearance, clothing, and action classes (pap, pcl, and pac), respectively. We employ the cross-entropy loss function to train the prediction of commonsense knowledge.

heading: 4.2 Graph-aware Symbolic Reasoning
text: The prediction for the action chain A requires rigorous layer-by-layer logical reasoning ability. Considering the symbolic network’s reasoning ability (Yi et al., 2018; Li et al., 2020c), we design a traditional graph theory based symbolic module for searching targeted nodes (actions).
In the training phase, we divide the action chain annotation labeled for each training example into several single-step action mappings and store them in the action relation memory module. During the process, new actions are continuously introduced. We initialize the prototype feature fip for the newly added action category with index i. In addition, at each step of the relation memory update, we construct the action graph based on the action relations. There may be Nd different connected paths between two nodes, which may result in the predicted action chain A not unique. To address the problem, we replace the starting node fsp with Nd different nodes {fsjp }Ndj=1 and view them as the starting of each path.
During the inference process, the action node described by the textual observation O is detected from the action graph. Using the traditional graph theory algorithm, Dijkstra (Dijkstra, 1959), we find all paths ending at this node. Assuming that there are Ns nodes on these paths, we view them as the candidate starting nodes and calculate the probability of being selected psn = {pisn}Nsi=1, using the transformer decoder predicted feature f2v:
psn = softmax([S(f1p, f2v),S(f2p, f2v), ...,S(fNsp , f2v)]), (8) where the S(.) is the similarity calculation function. The node with max probability is chosen out from the candidate starting nodes. With the starting and ending nodes, the actions corresponding to the middle nodes between them constitute the intermediate action chain A.

heading: 5 Experiments
text: We evaluate the effectiveness of the proposed NOVEL on TO-MAR dataset, followed by a discus-
sion of NOVEL’s property with controlled studies.

heading: 5.1 MAR Experiments
text: Implement Detail. Our model is implemented based on the PyTorch framework, which is trained on a Linux server. The implementation of the transformer part is based on the TubeDETR (Yang et al., 2022). For the training data, we randomly rotate and resize the input frames. In addition, random horizontal flips and size cropping are applied during the video frame preprocessing. For the validation and test data, we only normalize and randomly resize each frame. In the training process, the batch size is 1 and the random seed is 42. The learning rate is set to 0.00005 and the weight decay is 0.0001. All experimental environments are deployed in Hikvision (https://www.hikvision.com/en/).
Evaluation Metrics. Following the evaluation protocols of the spatio-temporal grounding (Su et al., 2021), we adopt m_vIoU and vIoU@R to evaluate the model performance. The vIoU is calculated by 1|Sp∪Sgt| ∑ n∈Sp∩Sgt IoU(b̂t, bt). In it, the Sp and the Sgt are the frame sets in the predicted and ground truth tubes, respectively. The b̂t and the bt are the predicted and ground truth bounding boxes of the frame t. The vIoU@R is the ratio of samples whose vIoU>R. The m_vIoU is the mean vIoU of all samples. In addition, we adopt the action chain accuracy (ACC) to evaluate the model performance for the action chain prediction.
Baselines. Existing methods of other tasks cannot be transferred directly to our MAR task. Thus, we extend several SOTA multi-modal and reasoning models as the baselines to compare. In detail, for a comprehensive comparison, we consider: (1) multi-modal video grounding methods, including TubeDETR (Yang et al., 2022), IT-OS (Li et al., 2022a), and STVGBert (Su et al., 2021); (2) action chain prediction methods, Cycle_C (Farha et al., 2020) and FUTR (Gong et al., 2022).
Performance Comparison. We compare our NOVEL model with the baselines on the TO-MAR dataset for the MAR task. The experiment results are shown in Table 2. From it, we can observe that our NOVEL model performs better than all previous methods. Specifically, compared to the previous state-of-the-art, TubeDETR, the NOVEL significantly improves the target event grounding (vIoU@0.3) from 18.7 to 24.4. In addition, the NOVEL model improves the action chain prediction from 62.1 to 72.0 compared with the best performance baseline, FUTR. We attribute the performance improvement of our model to commonsense knowledge-driven perception design. It helps the model focus on the correct visual semantics aligned with the textual observation. The graph symbol model rigorously describes the logical relationship between actions, and Dijkstra’s algorithm accurately reasons out the action chain.
Comparison using Different Training Data Volumes. We are interested in how the NOVEL model performance varies with the amount of training data. To this end, we randomly select different proportions of examples from the training set and compare our NOVEL model with several state-ofthe-arts trained on them. The experimental results are shown in Figure 4. From it, we can observe that the NOVEL model performance is always the best under different data volumes. Based on the commonsense knowledge recognition ability, the NOVEL model can eliminate the interference of irrelevant information on training, so that the model can learn the target video event grounding more effectively. Even if the training set is small, the NOVEL model still has higher accuracy.
Ablation Study. To fully evaluate the NOVEL model’s effectiveness, we need to understand how different components contribute. The new architectures are constructed by removing several compo-
nents from the NOVEL. The investigated building blocks include the knowledge-guided alignment and the graph-aware symbolic reasoning module. For convenience, we use ∆kno and ∆sym to represent these key components. After removing, the state-of-the-art model, FUTR, compensates for the lack of action chain prediction functionality.
The experiment results are shown in Table 3. From it, we can observe the following: (1) We evaluate our ablation grounding models and the FUTR model training together or not (labeled without or with * in the table). Notably, when training separately, the ∆kno is bound to the ablation grounding models, to prove its contribution to the NOVEL’s module. When training together, the pure neural network models fail to effectively capture the correlation between the target video event grounding and the action chain prediction. Even in the training process, the learning of the two interferes with each other, which leads to a loss of precision. (2) The NOVEL model performs better after adding each building block. It reveals the reasonable design of these key modules. (3) When the knowledgeguided alignment module and the graph-aware symbolic module are used together, the performance of the action chain reasoning is better. The results demonstrate the knowledge-guided alignment design aids in the extraction of knowledge useful for reasoning, and the symbolic reasoning module can complete accurate reasoning on this basis.
Case Study. A case study is conducted to demonstrate the NOVEL’s capability in visuals. In detail, two examples are sampled from our TOMAR dataset. A comparison of the NOVEL model with the state-of-the-arts, TubeDETR and FUTR, is necessary to fully demonstrate model performance on these examples. The experiment results are visualized in Figure 5. From the figure, we can find that our NOVEL model predicts the target video
event and the action chain accurately. In addition, its commonsense knowledge prediction is also correct. In contrast, the baseline predictions are not as satisfactory. This intuitively reflects that our neural-symbolic model, NOVEL, is reasonably designed for the multi-modal action chain Abductive Reasoning task.

heading: 5.2 Spatio-Temporal Grounding Experiments
text: Based on our proposed TO-MAR dataset, the models suitable for another similar language-vision understanding task, Multi-modal Spatio-Temporal Grounding (MSTG), can be evaluated. This task aims to detect the spatio-temporal tube described by the concise language sentence from the complex video content (Su et al., 2021). We further evaluate the effectiveness of the knowledge-guided alignment in our NOVEL model on this task.
Dataset. 9, 143 labels for this task based on the TO-MAR dataset are annotated by annota-
tors. Each label contains a natural language query, video clip temporal boundaries, and target object bounding boxes. We split all examples into 7, 845/277/1, 021 (train/val/test) without overlap and name this dataset as TO-MSTG. We describe the annotation method in detail in the appendix. In the future, we will further expand the data scale.
Performance Comparison. The experiment results are shown in Table 4. From it, we can observe that our NOVEL model performs best compared with the other three baselines. Specifically, it improves the accuracy (m_vIoU/vIoU@0.3/vIoU@0.5) from 8.9/9.5/4.5 to 11.7/12.0/7.0. This again demonstrates the power of commonsense knowledge guidance for the heterogeneous information alignment problem. In addition, the knowledge-guided alignment design generalizes effectively to different AI tasks, where heterogeneous alignment problem exists.

heading: 6 Conclusion
text: In this paper, we propose a new task, multi-modal action chain abductive reasoning, to promote the development of the abductive field. This cross-modal task targets to reason out a more complete explanation (explanation event grounding and sequential action chain inference) than the previous abductive reasoning tasks. Furthermore, we propose a largescale dataset (TO-MAR) and a neural-symbolic model via commonsense knowledge (NOVEL) for our new task as a strong baseline. Extensive experiments on the TO-MAR dataset and the TOMSTG dataset demonstrate the effectiveness of our NOVEL model.
Limitations
This work is currently limited to the action chain as the abstract summary of the complete explanation for the given limited observation. In the future, we will further upgrade this task, e.g., considering the progressive textual descriptions as the complete explanation. We hope our work can advance the
reasoning AI system research community.

heading: 7 Acknowledgments
text: This work is supported by the National Natural Science Foundation of China under Grant No. 62037001, the National Key RD Program of China under Grant No.2022ZD0162000, the National Natural Science Foundation of China under Grant No. 62222211 and Grant No.61836002. In addition, our research is funded by the Starry Night Science Fund at Shanghai Institute for Advanced Study (Zhejiang University).

heading: A For every submission:
text: A1. Did you describe the limitations of your work?
Not applicable. Left blank.
A2. Did you discuss any potential risks of your work? Not applicable. Left blank.
3 A3. Do the abstract and introduction summarize the paper’s main claims? section 1
A4. Have you used AI writing assistants when working on this paper? Not applicable. Left blank.

heading: B Did you use or create scientific artifacts?
text: Not applicable. Left blank.
B1. Did you cite the creators of artifacts you used? Not applicable. Left blank.
B2. Did you discuss the license or terms for use and / or distribution of any artifacts? Not applicable. Left blank.
B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? Not applicable. Left blank.
B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? Not applicable. Left blank.
B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? Not applicable. Left blank.
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. Section 3
C 3 Did you run computational experiments? Section 5
C1. Did you report the number of parameters in the models used, the total computational budget (e.g., GPU hours), and computing infrastructure used? Not applicable. Left blank.
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.
C2. Did you discuss the experimental setup, including hyperparameter search and best-found hyperparameter values? Not applicable. Left blank.
C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? Not applicable. Left blank.
3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Section 5
D 3 Did you use human annotators (e.g., crowdworkers) or research with human participants? Section 3
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? Not applicable. Left blank.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? Not applicable. Left blank.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? Not applicable. Left blank.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? Not applicable. Left blank.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? Not applicable. Left blank.

heading: Future Work
text: In the future, we will further upgrade this task, e. g. , considering the progressive textual descriptions as the complete explanation. We hope our work can advance the
reasoning AI system research community