Cross-modal Observation Hypothesis Inference

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-08 17:06:48

Abstract: Hypothesis inference, a sophisticated cognitive process that allows humans to construct plausible explanations for incomplete observations, is paramount to our ability to make sense of the world around us. Despite the universality of this skill, it remains under-explored within the context of multi-modal AI, which necessitates analyzing observation, recalling information in the mind, and generating explanations. In this work, we propose the Cross-modal Observation hypothesIs iNference task (COIN). Given a textual description of a partially observed event, COIN strives to recall the most probable event from the visual mind (video pool), and infer the subsequent action flow connecting the visual mind event and the observed textural event. To advance the development of this field, we propose a large-scale text-video dataset, Tex-COIN, that contains 39,796 meticulously annotated hypothesis inference examples and auxiliary commonsense knowledge (appearance, clothing, action, etc.) for key video characters. Based on the proposed Tex-COIN dataset, we design a strong baseline, COINNet, which features two perspectives: 1) aligning temporally displaced textual observations with target videos via transformer-based multi-task learning, and 2) inferring the action flow with non-parametric graph-based inference grounded in graph theory. Extensive experiments on the Tex-COIN dataset validate the effectiveness of our COINNet by significantly outperforming the state-of-the-arts.

Main paper: 
 introduction: 
 Hypothesis inference aims at proposing the most probable hypotheses to elucidate incomplete observation, which encapsulates a central pillar of human cognitive abilities to understand our surroundings [43] . This form of reasoning presents a dynamic interplay of memory and inference, where the incomplete event observation triggers the recall of historical visual events in memory, fostering the gradual inference of a holistic hypothesis serving as the explanation. For example, when we observe O: A burned boy is being treated in the hospital. and visually recall that he previously cooked in an oven, we can gradually infer a complete event chain as the explanation E: the boy cooked in an oven → he was burned → he put down the food → he cooled the wound → he hurriedly ran to the hospital. Hypothesis inference, with its wide-ranging applications in human existence, enables us to perform better than machines in high-level reasoning and would be the most precious capacity for modern AI (e.g., Judicial AI, Security AI). Yet, to the best of our knowledge, its integration into the realm of multi-modal AI systems -systems that strive to understand incomplete textual observations, visually recall related events, and infer the comprehensive chain of eventsremains a frontier largely unexplored.To facilitate the development in this AI field, we propose a novel task, Cross-modal Observation hypothesIs iNference (COIN). COIN draws its inspiration from the burgeoning advancements in the field of causal reasoning within the NLP community [3, 14] and further extends its reach into the multi-modal domain. The essence of COIN lies in its ability to recall the inception of an event from memory and infer a cohesive event flow that furnishes a robust explanation for an incomplete observation. Specifically, considering that 83% of the information memorized by the human brain is in the continuous visual form [46] , we integrate a video pool to simulate human visual memory. The AI system we propose is tasked with not only retrieving the target video that serves as the event chain's starting point but also inferring the subsequent action flow (subsequent event inference) that emerges after this retrieved video event and before the observed textural event. Examining and adopting ideas for a relevant, well-established, yet distinct task, i.e. text-video retrieval, provides a robust foundation for the development of this new field. The COIN task introduces two key distinctive elements that lend it a unique complexity: (1) Heterogeneous Alignment for Inference. Information alignment is the foundation of solving the retrieval tasks such as our COIN task and the text-video retrieval task. However, in the context of the COIN task, the textual observations and the target video do not align temporally, nor do they correspond to the same event. Instead, they describe logically correlated events, creating a challenging yet intriguing landscape for cross-modal information alignment. The task complexity is further amplified when we consider the need for extracting observation-related information (e.g., actions) in the video. (2) Action Flow Inference. There are rigorous logical connections between actions in the action flow. An error in the inference of any single action could potentially disrupt the correctness of subsequent actions. Action flow inference necessitates high fidelity in uncovering the relationships between actions and accurately predicting each one.To promote the development of this field, we propose a new large-scale dataset, Tex-COIN, towards the two distinctive characteristics of the COIN task. The dataset consists of about 10, 000 carefully collected videos, with 39, 796 hypothesis inference samples meticulously annotated by annotators with strong logical abilities. Our Tex-COIN dataset contains the following targeted designs: (1) burn put down cool run toThe burned boy with a gray sweater is being treated in the hospital. Commonsense Knowledge Annotation. We have incorporated a diverse range of commonsense knowledge annotations pertaining to the video characters described by the textural observation. These annotations, covering aspects such as appearance, attire, actions, and emotional state, are aimed at enhancing the model's heterogeneous alignment of retrieved video and textural observation.(2) Action Flow Annotation. For each video, annotators with notable logical capabilities are entrusted with annotating the incomplete textural observation, which envisions a future event, and the sequence of actions that transpire post the video event and preceding the textual event. In addition, the Tex-COIN dataset has the potential to advance broader task evaluations, such as temporally displaced text-to-video retrieval, thereby fostering deeper understanding and development in this area.Based on the carefully constructed Tex-COIN dataset, we propose a strong baseline, COINNet, hinged on commonsense knowledge and non-parametric inference for cross-modal observation hypothesis inference. There are two proposed modules targeted for the characteristics of the COIN task: (1) Knowledge-guided Cross-modal Alignment. We adopt the supervised multi-task learning approach, which facilitates the model's understanding of commonsense knowledge pertaining to the video characters as described in the textual observation. This includes the alignment of heterogeneous visual information with the textual observation, such as character actions and scenes that are closely connected but temporally displaced from the textual observation. This perception of visual character actions lays the foundation for effective inference of the action flow. (2) Graphbased Non-parametric Inference. We develop a non-parametric action flow prediction module grounded in the principles of traditional graph theory. Specifically, during the training phase, we store the relationships between actions. In the testing step, we construct an action graph with the stored action relationships and apply the Dijkstra's algorithm [10] to locate the path connecting the actions described in the video and the textural observation. The experiments on the Tex-COIN dataset prove the efficacy of the COINNet model by significantly surpassing the state-of-the-arts.Our contributions can be summarized as follows:• We propose the Cross-modal Observation Hypothesis Inference task (COIN) by simulating human cognition. To the best of our knowledge, it is the early exploration of hypothesis inference in the multi-modal field.  
related work: 
 Multi-modal Inference With the development of the AI system [1, 9, 11, 28, 33] , there are tremendous breakthroughs in the completion of perception tasks. However, further advancements are needed to complete multi-modal hypothesis inference tasks using AI systems. Lots of researchers are involved in this process [2, 6, 7, 18, 22, 31, 52] . In the area of the multi-modal hypothesis inference, some researchers introduce knowledge into auxiliary reasoning to complete [5, 8, 48, 56, 57] . [48] involves the external knowledge in the process of the visual question answer and achieves good results. In addition, lots of researchers focus on the fusion of cross-modal information [4, 17, 19, 32, 37] . [37] joints analysis of the input question and the input video with the convolutional graph to complete the video question answering task. Aligning the crossmodal information is widely adopted in the multi-modal inference domain [12, 23, 30, 54] . [12] aligns the cross-modal features with relevance affinity matrix. Compared with previous tasks, our COIN task has two different characteristics: (1) Our COIN task requires retrieving the target video from the video pool to explain the observation;(2) COIN needs to rigorously recover the complete action flow. They determine the distinctiveness of our Tex-COIN dataset, targeted for our task.Cross-modal Transformer. Transformer-based models are becoming the mainstream algorithms of more and more cross-modal tasks [26, 27, 29, 50] . [44] introduces the transformer architecture into the representation learning of the text and image. Many researchers [35, 36] bring the pre-trained model into different tasks (like text-video grounding) and achieve good performance. [55] decouples the spatio-temporal reasoning of the video effectively reducing the number of parameters inside the transformer. Similarly, [50] adopts the transformer model to complete the cross-modal spatial-temporal grounding task and achieves good performance. [55] introduces the parameters of the pre-trained image-text grounding model into its transformer structure and improves the model performance. 
hypothesis inference dataset: 
 The Cross-modal Observation hypothesIs iNference task (COIN) aims to explain the textual observation by predicting an event chain (cause), which contains two sub-tasks: (A) retrieving the video from a video pool as the event chain starting point; (B) predicting the subsequent action flow after the retrieved video to complete the event chain. Towards two subtasks, there are two characteristics of our COIN task: Heterogeneous Alignment for Inference and Action Flow Inference. To promote the development of this field, we propose a large-scale dataset, Tex-COIN, which contains the commonsense knowledge annotation assisted to the cross-modal alignment training and the hypothesis inference example annotation. In this section, we will describe it in detail.Task Formulation. Our Tex-COIN dataset is designed specifically for the characteristics of the COIN task. Thus, it is necessary to formulate the COIN task, before introducing our Tex-COIN. Given a textual observation (a language query) O and a video pool between the target video V 𝑡 and the textual observation O. With the AI system for our COIN task denoted as M with the parameter Θ, the optimization function 𝛿 of the AI system M can be expressed as follows:V = {V 𝑖 } 𝑁 V 𝑖=1 containing 𝑁 V videos,EQUATIONIn it, Θ is a learnable parameter. The AI system M (.) generates the model prediction and the function 𝜖 (.) outputs the ground truth.Then, the function 𝜉 (.) calculates the consistency of model prediction and the ground truth, which represents the loss calculation process described in Section 4.1. Dataset Construction Pipeline. As shown in Figure 2 , we summarize the dataset construction pipeline into three steps: (A) Data Collection. We collect the videos used for labeling from two sources, including lifestyle videos and TV show videos. Further, the annotators screen out the unqualified videos one by one. We add the detail to the appendix. (B) Commonsense Knowledge Annotation. The annotators label the commonsense knowledge (appearance, clothing, action, etc.) of the key characters in the collected videos. Then, the verifiers carefully check all the labels. (C) COIN Task Annotation. The labeling process for the COIN Task Annotation is similar to the Commonsense Knowledge labeling and consists of manual labeling and verification. In addition, we add additional annotations for the text-video retrieval task to widen the dataset function, which is introduced in the appendix.Data Collection. Before illustrating the annotation process, we introduce how to collect the data in our Tex-COIN. Specifically, The data in our Tex-COIN dataset comes from two sources: (A) We manually crop 10, 000 video clips from 92 well-known TV shows, including Mr. Bean, Grey's Anatomy, etc. In addition, we carefully select 10, 000 video clips from the previous datasets, including the HC-STVG dataset [51] , the ava-actions dataset [21] , and the TO-MAR dataset [34] . (B) 20, 000 lifestyle videos are carefully collected from the Hwd dataset [49] and the vidor dataset [47] . Not all the collected videos are suitable for annotation in the next step, like the videos with extremely poor clarity, the videos with frequent transitions between story segments, the videos with long, meaningless static shots, etc. Thus, to construct the high-quality dataset, we carefully filter out the low-quality videos before annotation. Commonsense Knowledge Annotation. With the filtered videos, we annotate the commonsense knowledge of the key characters in each video, including the characters' appearance, clothing, actions, sentiments, and located scenes. There are many subcategories in each type of commonsense knowledge, as shown in Table 1 . To ensure high-quality labels, we divide the entire process into two parts, annotation and verification: (A) Annotation. Considering a large number of detailed labels required manual annotation, we organize a team of 20 annotators to complete the labeling. Among them, one annotator is responsible for the management, and 19 annotators are responsible for labeling. The entire labeling process takes 10 months to complete. (B) Verification. 4 verifiers are responsible for the verification of each annotated example, if all of them agree with the annotation, the example is saved. Otherwise, it is re-labeled and discarded.COIN Task Annotation. Based on the filtered videos, The labeled examples that have unanimous agreement are accepted.In addition, the language-based retrieval uniqueness needs to be guaranteed, which is added to the appendix.Dataset characteristics We are interested in the characteristics of the dataset. Therefore, we conduct a detailed analysis of the dataset, which is summarized as follows: (A) Large-scale. We propose a large-scale dataset for the COIN task, which consists of 39,796 hypothesis inference examples. We randomly split it into the nonoverlapping train/val/test set (26,904/6,000/6,892). (B) Diversity.There are two types of data sources, including real daily-life data sources and TV show data sources. We manually filter out the highquality data to guarantee that there are enough diverse people, actions, and scenes. At the same time, we also try our best to ensure that the logical chains in the videos are clear and diverse in the process of data selection. Dataset Statistics. We conduct a statistical analysis of our Tex-COIN. The length statistics of the natural language queries (Figure 3 ) highlight the diverse amount of information conveyed through different language queries. In addition, from the statistics of the videos with different action numbers (Figure 3 ), there are significant variations in the video action number. It reflects that our dataset covers videos with varying levels of content richness (action richness). We add more statistical results and analysis to the supplementary. 
method: 
 Based on the novel task, Cross-modal Observation hypothesIs iNference (COIN), we introduce a carefully designed strong baseline, COIN-Net, whose diagram is shown in Figure 4 . In this section, we will describe our COINNet in detail.Model Pipeline. Our COINNet judges whether the given video V 𝑖 is part of the explanation of the textual observation O. Then, we choose the most suitable video with the highest probability from the video pool V and output the subsequent action flow as the explanation of the textual observation O. In detail, the model pipeline consists of three steps: The alignment of the cross-modal information is fundamental to textvideo retrieval [24, 40] , which is an important subtask of our COIN. Thus, we propose the targeted knowledge-guided cross-modal alignment module. This module is trained in a supervised manner and could precisely predict the key commonsense knowledge (appearance, clothing, action, etc.) of the target video person described by the textual observation O.Specifically, the knowledge-guided cross-modal alignment module is designed based on the transformer model [53] . The video feature F 𝑣 is extracted from the given video V 𝑖 and the language feature F 𝑡 is extracted from the textual observation O, with the visual encoder [13] and the language encoder [53] . Then, two types of query vectors are defined to analyze the multi-modal feature F 𝑣The burned boy with a gray sweater is being treated in the hospital.Frame-Text Level Semantic SceneFrame-Text Level Feature Video-Text Level Feature Information Flow Action RetrieveStep 1Step 2Step 3 Q 𝑣 = {q 𝑖 𝑣 } 3 𝑖=1 .𝑁 𝑓 is the number of frames in the video V 𝑖 . We inject the type embeddings to help the transformer encoder distinguish different types of query vectors. Then, the transformer encoder reasons out the corresponding features (frame-text level featuresF 𝑓 = {f 𝑖 𝑓 } 𝑁 𝑓𝑖=1 and video-text level features F 𝑣 = {f 𝑖 𝑣 } 3 𝑖=1 ) relying on the corresponding query vectors (Q 𝑓 and Q 𝑣 ). After analyzing features output by the transformer encoder (F 𝑓 and F 𝑣 ), we predict all commonsense knowledge.(1) Sentiment and Scene Prediction. Considering that the sentiment knowledge and the scene knowledge are changed over time they are predicted frame by frame with the frame-level features F 𝑓 . We use the 𝑖-th frame as an example, and then the prediction process for the 𝑖-th frame with the corresponding frame feature f 𝑖 𝑓 denoted as:EQUATIONEQUATIONIn them, 𝑀𝐿𝑃 𝑠𝑒 and 𝑀𝐿𝑃 𝑠𝑐 are the MultiLayer Perceptron (MLP), which are applied to predict the sentiment and scene probabilities (p 𝑖 𝑠𝑐 and p 𝑖 𝑠𝑒 ), respectively. (2) Appearance and Clothing Prediction. Regarding the appearance and clothing knowledge of the target person in the video, they are approximated as remaining constant throughout the video and are applied with the video-level query f 2 𝑣 to predict. In rare cases where there are special situations (such as costume changes), we require the model to predict the most relevant appearance and clothing knowledge for the textual observation description. We can represent the prediction process for the appearance and clothing knowledge as follows:EQUATIONEQUATIONIn the process, 𝑀𝐿𝑃 𝑎𝑝 and 𝑀𝐿𝑃 𝑐𝑙 are the MLP utilized to predict the appearance and clothing probabilities (p 𝑎𝑝 and p 𝑐𝑙 ), respectively.(3) Action Prediction. When it comes to action knowledge, we also do not make frame-by-frame predictions as we do for sentiment knowledge and scene knowledge. It is due to the presence of multiple actions for the target person in each frame. Making frame-by-frame predictions would impose a heavy burden on our model and make it difficult to train effectively. As a result, the model directly predicts the target action most related to the textual observation O, instead of counting all video actions for each frame individually. We can represent the process of predicting the target video action using the video-level query feature f 1 𝑣 as follows:EQUATIONIn it, 𝑀𝐿𝑃 𝑎𝑐 is the MLP used for the target action prediction. p 𝑎𝑐 represents the action type probability of the target action.The model learns to make all commonsense knowledge predictions simultaneously. To achieve this, we use the cross-entropy loss function to calculate the losses for all commonsense knowledge, including 𝑙 𝑠𝑒 (sentiment), 𝑙 𝑠𝑐 (scene), 𝑙 𝑎𝑐 (action), 𝑙 𝑎𝑝 (appearance), and 𝑙 𝑐𝑙 (clothing). To obtain the complete knowledge-guided loss 𝑙 𝑡𝑜𝑡𝑎𝑙 , we sum up all the individual losses mentioned above:EQUATION)Due to the interdependent and interconnected actions in the action flow, the action flow inference subtask requires high correctness in learning the relationship between the actions and precisely predicting each action. Thus, we propose the targeted module, Graph-based Non-parametric Inference, which stores the action relationships to construct the action knowledge graph and finds the target action flow from the constructed graph with Dijkstra's algorithm. Specifically, we represent the action set asA = {A 𝑖 } 𝑁 A 𝑖=1. The module diagram is shown in Figure 4 . From it, we can find that there are two steps in the graph-based non-parametric inference module:(1) Training Step. During the training process, we 𝑠𝑝𝑙𝑖𝑡 the action flow labeled for each example into multiple single-step relational maps and store them in the action memory. No trainable parameters are involved in this process and we don't need to further train the action memory module. We take the action flow A 1 → A 2 ... → A 𝑟 as an example to present this process, which is represented as:EQUATION)(2) Testing Step. At the testing step, the action knowledge graph G(A, U) is constructed by 𝑐𝑜𝑛𝑛𝑒𝑐𝑡𝑖𝑛𝑔 all the single-step relational maps (A 𝑖 → A 𝑖+1 ), ..., (A 𝑖+𝑛-1 → A 𝑖+𝑛 ) in the memory generated in the step (1) . Here, we define U as the edges between the actions in the action knowledge graph. With the constructed action knowledge graph, we find the target path as the predicted action flow during the testing process. Firstly, the starting action knowledge node A 𝑠 and the ending action knowledge node A 𝑒 of the target path (target action flow) are found in the constructed graph G(A, U). We find the starting node A 𝑠 by matching the action predicted by the knowledge-guided cross-modal alignment module (described in Section 4.1) and each action knowledge node in the graph G(A, U). Similarly, the ending node A 𝑒 is found by matching each action knowledge node in the graph G(A, U) and the action of the textual observation O. The textual action is detected with the widely applied tool, StanfordNLP [41] . Secondly, we use the Dijkstra's algorithm to find the connected path between the starting node A 𝑠 and the ending node A 𝑒 . We formalize the path-finding process as:EQUATIONAfter finding the connected path, the found action flow (connected path) is preserved and needs to be further verified in the next reasoning path review module.As the key module in our COINNet model, this module reasons out the matching score between the given video V and the textual observation O, according to the cross-modal feature f 0 𝑣 and the predicted action flow A 𝑠 ... → A 𝑒 .Specifically, we first encode the found action flow A 𝑠 ... → A 𝑒 , with a new transformer encoder different from the one in the Knowledge-guided Cross-modal Alignment module. Then, we get the feature representation f A of the action flow. We predict the crossmodal matching score with the video-text feature f 0 𝑣 (containing the information from the given video V and the textual observation O) and the action flow feature f A :EQUATIONwhere p 𝑚𝑎𝑡𝑐ℎ is the probability vector of the cross-modal matching or not.[.] represents the feature concatenation. In addition, 𝑀𝐿𝑃 𝑚𝑎𝑡𝑐ℎ (.) represents the multi-layer perception and softmax(.) is the activation function. If there are multiple found action flows in step 2 of our COINNet model, we select the one corresponding to a higher predicted matching score as the final result. 
experiment: 
 In this section, we will evaluate our proposed baseline COINNet on our Tex-COIN dataset.Baselines. Previous methods designed for other tasks cannot be applied to solve our COIN task, directly. Thus, we extend several state-of-the-arts to compare. In detail, for a comprehensive comparison, we introduce the following methods of similar format tasks:(1) text-video retrieval methods, including DMAE [25] , MASCOT [15] , CRET [24] , Clip4Clip [40] (2) action flow prediction methods, including CycleC [16] , FUTR [20] , and VLMAH [42] . During the performance comparison, we pairwise fuse them to form 10 baselines: DMAE_C (DMAE + CycleC), DMAE_F (DMAE+FUTR), MASCOT_C (MASCOT + CycleC), MASCOT_F (MASCOT+FUTR), MASCOT_V (MASCOT+VLMAH), CRET_C (CRET + CycleC), Clip4Clip_C (Clip4Clip + CycleC), CRET_F (CRET + FUTR), Clip4Clip_F (Clip4Clip + FUTR), Clip4Clip_V (Clip4Clip + VLMAH).Implement Detail. We implement our COINNet on a Linux server with Pytorch 1.4 and 8 Tesla V100 with 32GB memory. During the training process, we set the training rate as 1𝑒 -5. The batch size is set to 8 and the optimizer is AdamW. In addition, we also employ some data augmentations, such as random horizontal flipping, random cropping, and so on. We adopt the same pretraining parameters for the COINNet as the Clip4Clip baseline [40] .Evaluation Metrics. Following the widely used evaluation protocol of the text-video retrieval task [24, 39] , Rank N (R@N, N=1,5,10) and median rank (MdR) are applied to evaluate the model performance for the explanation video retrieval. We adopt MoC used by the future action prediction tasks [16] to evaluate the action flow predicted results In addition, we also apply accuracy (ACC) to evaluate the model performance for the action flow prediction, which is added in the appendix.R@1 R@5 R@10 MdR MoC DMAE_C We are interested in the COINNet performance compared with the state-of-the-arts. The comparison results are shown in Table 2 . From it, we have the following findings:• It can be found that our COINNet model performs better than other baselines. We attribute the performance improvement to the cross-modal precise alignment with the knowledgeguidance, and the action-relation learning and inference ability of the Graph-based Non-parametric module. • The baselines, like DMAE_C and CRET_C, are extended from other baselines and lack the domain knowledge, which limits the performance of these baselines.In addition, we fully compare our COINNet with the generative large language model in the appendix, which proves the same conclusion.We are interested in the effect of each COINNet module. Thus, we design the ablation study by surgically removing these modules,R@1 R@5 R@ In addition, we add the guidance of each commonsense knowledge independently (including appearance, clothing, action, sentiment, and scene), to evaluate the effectiveness of each one. The experiment results are shown in Table 4 . From it, we can observe that the model performs better after adding each of the commonsense knowledge guidance. It proves the reasonable design of the Knowledge-guided Cross-modal Alignment. Finally, we analyze the COINNet performance solely relying on the character's physical description to retrieve the target video and add the experiment results to the appendix. proposed modules, the accuracy of COINNet remains higher than that of the Clip4Clip_F baseline, even with a smaller training data volume.Hyperparameter Analysis We are interested in the effectiveness of the training hyperparameters. Thus, we select the key hyperparameter, the learning rate, for further analysis. The experiment results are shown in Figure 5(b) . Notably, the COINNet model achieves the highest accuracy with a learning rate of 1e-4, which is adopted by us. In addition, our method, COINNet, has low sensitivity to learning rate, which reflects its robustness.Case Study To further demonstrate the effectiveness of our COIN-Net, we randomly select an example from the Tex-COIN dataset and visualize the prediction of our COINNet baseline. The experiment results are shown in Figure 6 . In the shown case, the complete event process is: The woman cooked in the kitchen. → She ate the food. → She cleaned the kitchen waste. → She picked up, held, and threw the food waste. It can be observed that our strong baseline, COINNet, precisely predicts the target video and the action flow, and reasons out all commonsense knowledge correctly, which proves the reasonable design of our COINNet. More examples are shown in the appendix.We add the text-video retrieval annotations for our Tex-COIN dataset to extend the dataset function, and experiment with the retrieval baselines on the Tex-COIN dataset. In this section, we will describe them in detail.R@1 R@5 R@10 MdR Performance Comparison. We compare our COINNet model with the state-of-the-art on Tex-COIN dataset with the text-video retrieval annotations. In addition, we conduct the ablation study for our COINNet. The experiment results are shown in Table 5 . From it, it can be found that our COINNet model performs best. We attribute the improvement to the effectiveness of the Knowledgeguided Cross-modal Alignment design. 
conclusion: 
 In this paper, we propose a new task (COIN), the task-toward dataset (Tex-COIN), and the targeted strong baseline (COINNet), to promote the improvement of hypothesis inference in the multi-modal field. Experiments on the Tex-COIN prove the effectiveness of our COINNet model. We believe that our work can assist in building the human-like reasoning AI system and help to improve its performance in hypothesis-inference applications. For instance, in intelligent security, given the description of the witness, AI system with the hypothesis inference capability can search for visual evidence from surveillance and infer the potential criminal process. Although some progress has been made, there are still several limitations left for future work: (1) The model accuracy is far from reaching its upper limit. (2) Following the previous datasets [38, 45] , we collect data from the real-life and movie domains. More data from other domains can further enrich our dataset. In addition, we believe that our work can assist in building the human-like reasoning AI system and help to improve its performance in hypothesis-inference applications. For instance, in intelligent security, given the description of the witness, the AI system with the hypothesis inference capability can search for visual evidence from surveillance and infer the potential criminal process.  
 
 Appendix:  


Decision: All reviewers have expressed positive attitudes towards this paper, so we have decided to accept this paper.

Reviews: review: paper_summary: paper_summary: This study proposes a cross-modal observation hypothesIs inference task named COIN, aiming to  recall the inception of an event from memory and infer a cohesive event flow that furnishes a robust explanation for an incomplete observation.
Based on the task, the study constructs a Tex-COIN dataset and introduces COINNet as baseline.
strength_weakness: strength_weakness: 1. The cross-modal observation hypothesIs inference task is interesting and challenging. The hypothesIs inference capacity is valuable for modern AI. Heterogeneous alignment for inference and action flow inference make the task challenging.

2. The Tex-COIN dataset is well-contructed. The diagram of the dataset construction pipeline and the  statistics of dataset are clear.

3. The baseline for the COIN task is given named COINNet.
limitations: limitations: 1. The evaluation criteria for this task are worth discussing. 

Since it is motivated as a inference task, the evaluation of the inference results should be subjective. However, the widely used evaluation protocol of the text-video retrieval task and accuracy of prediction task  can only  reflect the performance of  model output matching with the dataset.

The potential issue is, if the model I propose can have good retrieval and predictive performance, then will the reasoning I output conform to human objective cognition, can it explain observations well, and even be logically coherent?
paper_summary: paper_summary: The paper introduces a new task called Cross-modal Observation Hypothesis Inference (COIN), which involves creating a system that can recall events from a visual memory to explain partially observed textual descriptions. The proposed system, COINNet, uses a novel dataset, Tex-COIN, containing text-video examples and integrates commonsense knowledge to enhance inference accuracy. Extensive experiments validate the effectiveness of COINNet, which significantly outperforms existing models.
strength_weakness: strength_weakness: 1. The COIN task is novel and taps into an underexplored area of multi-modal AI, merging hypothesis generation with visual and textual analysis.
2. The development of the Tex-COIN dataset, which includes diverse examples and commonsense annotations, provides a solid foundation for training and evaluating the model.
3. COINNet integrates advanced techniques like transformer-based multi-task learning and graph-based inference, demonstrating superior performance over state-of-the-art models in comprehensive tests.
limitations: limitations: 1. While the paper introduces a novel dataset and model, the generalizability of COINNet across different datasets and real-world scenarios remains unclear. The specialized nature of the Tex-COIN dataset might not represent the variety of contexts and environments where such a system would be deployed.
2. The development and training of COINNet potentially inherit biases present in the dataset. The paper does not address how it handles potential biases or the impact these biases could have on the model's performance.
3. A few minor errors. For example, with lines 426 and 677, the text content exceeds the limit. Please check the full text.
paper_summary: paper_summary: The paper introduces a novel task called Cross-modal Observation Hypothesis Inference (COIN), aiming to simulate human cognitive processes in multi-modal AI systems. The authors present a large-scale dataset, Tex-COIN, and a strong baseline model, COINNet, which integrates commonsense knowledge and non-parametric inference for cross-modal hypothesis inference. The method is validated on Tex-COIN. SOTA performance is achieved.
strength_weakness: strength_weakness: 1.  The paper introduces a new and challenging task focusing on cross-modal observation hypothesis inference through the alignment of textual observations with visual content. 

2. Experiments validate the superior performance of COINNet over existing state-of-the-art models.

3. The paper is well organized and easy to follow. The idea and the method are well presented.
limitations: limitations: 1.   The baseline comparisons may not be comprehensive enough to demonstrate the full efficacy of the proposed model.  The experiments on the Tex-COIN dataset focus primarily on comparing COINNet with state-of-the-art models **in a limited set of scenarios**. Expanding the scope to include more diverse and challenging benchmarks would strengthen the evaluation. And the detailed structure and parameters of other models should be contained for a fair comparison.

2. COINNet seems to primarily consider temporal relations among objects and descriptive questions within the video. However, deep inference capabilities, including causal reasoning, are crucial for comprehensive video understanding. Does the current task setup truly challenge these advanced inference abilities?


3. Could **the heavy reliance on manual annotations introduce biases and scalability issues**? The Tex-COIN dataset contains 39,796 meticulously annotated hypothesis inference examples by annotators with strong logical abilities. Might this reliance on manual annotation processes lead to inconsistencies and pose challenges for scaling the dataset effectively?


4. What about the complexity or time cost? Providing an analysis of the computational complexity and the time required for training and inference, compared to other methods, would offer valuable insights into the practical applicability of COINNet.