SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Baseline Model	Accuracy (%)
PACS	PACS-Material
Late Fusion [55]	55.0 ± 1.1	67.4 ± 1.5
Late Fusion w/ DCL	57.7 ± 0.9	69.7 ± 1.2
CLIP [56]	56.3 ± 0.7	72.4 ± 1.1
CLIP w/ DCL	58.4 ± 0.8	75.4 ± 1.2
AudioCLIP [57]	60.0 ± 0.9	75.9 ± 1.1
AudioCLIP w/ DCL	63.2 ± 0.8	76.2 ± 1.4
UNITER(Large) [58]	60.6 ± 2.2	75.0 ± 2.8
UNITER w/ DCL*	62.0 ± 2.4	75.7 ± 2.8
(a) Comparison of our DCL with different baselines.
Baseline Model	Accuracy (%)
PACS	PACS-Material
Late Fusion [55]	55.0 ± 1.1	67.4 ± 1.5
Late Fusion w/ MLP	54.9 ± 0.9	67.7 ± 1.1
Late Fusion w/ DSE	56.2 ± 0.8	68.5 ± 1.2
CLIP[56]	56.3 ± 0.7	72.4 ± 1.1
CLIP w/ MLP	56.5 ± 0.5	72.6 ± 1.2
CLIP w/ DSE	57.0 ± 0.6	73.2 ± 1.1
AudioCLIP[57]	60.0 ± 0.9	75.9 ± 1.1
AudioCLIP w/ MLP	60.3 ± 0.8	76.2 ± 1.3
AudioCLIP w/ DSE	61.1 ± 0.8	76.5 ± 1.0
(b) Ablation study of DSE.
Baseline Model	Accuracy (%)
PACS	PACS-Material
Late Fusion [55] w/ DSE	56.2 ± 0.8	68.5 ± 1.2
Late Fusion w/ DSE,A	57.0 ± 1.1	68.9 ± 1.3
Late Fusion w/ DSE,A,C	57.7 ± 0.9	69.7 ± 1.2
CLIP [56] w/ DSE	57.0 ± 0.6	73.2 ± 1.1
CLIP w/ DSE,A	57.8 ± 0.8	74.5 ± 1.1
CLIP w/ DSE,A,C	58.4 ± 0.8	75.4 ± 1.2
AudioCLIP [57] w/ DSE	61.1 ± 0.8	76.0 ± 1.0
AudioCLIP w/ DSE,A	61.9 ± 0.9	75.8 ± 1.1
AudioCLIP w/ DSE,A,C	63.2 ± 0.8	76.2 ± 1.4
(c) Ablation study of CLM
Baseline Model	Accuracy (%)
PACS-Material	Δ
Late Fusion [55] w/ dynamic	60.4 ± 1.3	-
Late Fusion w/ static	66.0 ± 0.6	5.6
Late Fusion w/ static, dynamic	69.7 ± 1.2	3.7
CLIP [56] w/ dynamic	63.0 ± 0.7	-
CLIP w/ static	71.1 ± 0.9	8.1
CLIP w/ static, dynamic	75.4 ± 1.2	4.3
AudioCLIP [56] w/ dynamic	67.1 ± 0.6	-
AudioCLIP w/ static	72.5 ± 1.0	5.4
AudioCLIP w/ static, dynamic	76.2 ± 1.4	3.7
(d) Ablation analysis of the static factors.
Table 1: Experimental results and analysis in terms of the accuracy achieved on PACS and PACS-Material. 'DSE' refers to Disentangled Sequential Encoder, while 'CLM' refers to Counterfactual Learning Module. 'A' represents physical knowledge relationship, and 'C' stands for Counterfactual Relation Intervention.  $\Delta$  represents the performance improvement compared with the previous row.
4.2 Comparison to Baselines
Quantitative Results. We report the quantitative performance comparisons using the PACS dataset in Table 1(a). The results show that with the addition of our proposed DCL, both the Late Fusion and UNITER models register absolute improvements of  $2.7\%$  and  $1.4\%$ , respectively. Similarly, CLIP and AudioCLIP models, which align the image, audio, and text into a shared space, achieve absolute improvements of  $2.1\%$  and  $3.2\%$ , respectively. These results highlight the strong reasoning and generalization capabilities of our DCL method.
Interestingly, even with the DCL, CLIP's performance trails that of AudioCLIP, emphasizing the critical role of audio information. When comparing CLIP and AudioCLIP enhanced with our DCL, it is evident that the inclusion of audio information leads to an absolute improvement of  $4.8\%$ . The same conclusion applies to the PACS-Material subset, indicating that our method, serving as a plug-and-play module, can consistently enhance material reasoning performance and be flexibly incorporated across various models. It is noteworthy that all objects in the test set were absent from the training and validation sets, attesting to the zero-shot reasoning ability of our model.
Qualitative Results. In order to provide a comprehensive qualitative perspective on our results, we present the visualized results of different models for the same questions in Figure 2. In Figure 2(a), the two objects exhibit significant visual and auditory differences, facilitating accurate inference by both the baseline model and our DCL. Conversely, the two objects in Figure 2(b) share similar auditory and static information but differ significantly in dynamic information. Our method accurately infers the answers to both questions. This is especially notable in the case of the second question, which requires dynamic information for reasoning, thus highlighting the efficacy of our DCL in the decoupling of static and dynamic elements.
Figure 2: Qualitative Results of baseline w/ and w/o our DCL, where "Material" refers to the material of the object, and "Properties" refers to the physical properties that the question focuses on. The correct answers are depicted in green while the incorrect ones are depicted in red.
In Figure 2(c), only AudioCLIP enhanced with our DCL accurately answers the first question, while the other models fail. This discrepancy arises from inaccurate labeling within the dataset, where two objects appear in the same video, and the bounding box appears to have been assigned to the wrong object. By integrating a more robust object detection model and fine-grained annotations, our proposed DCL can circumvent such issues.
In this section, we perform the ablation studies to examine the effectiveness of each of our proposed modules, including the Disentangled Sequential Encoder, physical knowledge relationship matrix, and counterfactual relation intervention in the Counterfactual Learning Module.
Disentangled Sequential Encoder (DSE). Table 1(b) shows the performance comparison of our proposed DSE with different baselines. And we can see AudioCLIP can achieve the best results. After incorporating DSE, all baselines are able to achieve significant improvements in PACS and PACS-Material. However, adding an MLP module with the same parameters as DSE did not yield such improvement, indicating that the proposed DSE can effectively enhance the ability of physical characteristic expression in video features.
Counterfactual Learning Module (CLM). Table 1(c) presents the ablative analysis of two designed parts in the Counterfactual Learning Module (CLM). After adding the physical knowledge relationship denoted as affinity matrix  $A$ , all baselines showed an improvement in performance. Similarly, the same result can be clearly found when adding the Counterfactual Relation Intervention denoted as 'C'. Additionally, the same conclusion can be drawn from the PACS-Material, demonstrating that both parts of our Counterfactual Learning Module can independently improve results and achieve optimal performance when used simultaneously.
4.4 The Analysis of Disentanglement Learning
In this section, we carry out additional experiments and analysis focusing on the decoupled static and dynamic factors obtained through the proposed Disentangled Sequential Encoder. Specifically, we utilize separated individual factors to predict the object's material information, excluding multi-modal
Figure 3: The t-SNE visualization of the obtained dynamic factors in the latent space of video samples in the test set. The points located close to each other as a cluster exhibit similar or identical actions, such as rubbing with the hand (left) and friction with the hand (right), while such two clusters are pulled away showing they have completely different dynamic information.
feature fusion. For the captured dynamic factors, we present the t-SNE visualization of various video samples to illustrate the distribution of these dynamic factors within the latent space.
Analysis of static factor. The material classification results are shown in Table 1(d), where "w/ static" and "w/ dynamic" respectively signify reasoning using only decoupled static factors and dynamic factors, while "w/ static, dynamic" denotes using both static and dynamic factors simultaneously. It can be observed that combining the static and dynamic factors significantly improves the accuracy of reasoning for material problems, compared to using only dynamic factors. This indicates the crucial importance of static factors in material property inference. When predicting using only static factors, there is no significant reduction in accuracy, confirming the adequacy of static factors in providing sufficient material-related information that is not captured by dynamic factors. This also implies that the static and dynamic factors contain little overlapping information and validate the effectiveness of our comparative estimation of mutual information (MI).
Analysis of dynamic factor. For dynamic factors, we decoupled the test set videos and showed the t-SNE visualization of the results, as shown in Figure 3. It can be observed that there is a certain clustering phenomenon in the latent representation of dynamic factors. By displaying the corresponding videos of the data points, we found that the points on the left panel of the figure mostly represent actions involving rubbing with objects by hand. In this case, the visual features are often insufficient to fully represent the characteristics of the object due to its thickness or weight, and the audio information is often the main basis for human reasoning. On the other hand, the points on the right side of the figure correspond to actions involving the friction of objects. Such actions can reveal more dynamic properties of the object, such as malleability. In this case, the dynamic information in the video is often the main basis for human reasoning. Through the above analysis, we can see that for different types of dynamic information, humans will also focus on different modalities of information. This is helpful for future improvements of our reasoning models.
In this paper, we introduced a method named Disentangled Counterfactual Learning (DCL) for physical audiovisual commonsense reasoning, in which a Disentangled Sequential Encoder was designed to decouple the video into static and dynamic factors represented as time-invariant and time-variaged information related to actions, respectively. Additionally, we model the physical knowledge relationship among different target objects as an affinity matrix and apply counterfactual relation intervention to guide the model to emphasize the physical commonalities among different objects. Designed as a plug-and-play component, our method can be readily incorporated and has demonstrated its potential to significantly enhance multiple baselines.
This work was partly supported by the Funds for Creative Research Groups of China under Grant 61921003, the National Natural Science Foundation of China under Grant 62202063, Beijing Natural Science Foundation (No.L223002), the Young Elite Scientists Sponsorship Program by China Association for Science and Technology (CAST) under Grant 2021QNRC001, the 111 Project under Grant B18008, the Open Project Program of State Key Laboratory of Virtual Reality Technology and Systems, Beihang University (No.VRLAB2022C01).
[1] Nikolaus Kriegeskorte. Deep neural networks: a new framework for modeling biological vision and brain information processing. Annual review of vision science, 1:417-446, 2015.
[2] Samuel Yu, Peter Wu, Paul Pu Liang, Ruslan Salakhutdinov, and Louis-Philippe Morency. Pacs: A dataset for physical audiovisual commonsense reasoning. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XXXVII, pages 292-309. Springer, 2022.
[3] Christopher J Conti, Aparna S Varde, and Weitian Wang. Robot action planning by commonsense knowledge in human-robot collaborative tasks. In 2020 IEEE International IOT, Electronics and Mechatronics Conference (IEMTRONICS), pages 1-7. IEEE, 2020.
[4] Jean-Luc Lugrin and Marc Cavazza. Making sense of virtual environments: action representation, grounding and common sense. In Proceedings of the 12th international conference on Intelligent user interfaces, pages 225-234, 2007.
[5] Senthil Purushwalkam, Sebastia Vicenc Amengual Gari, Vamsi Krishna Ithapu, Carl Schissler, Philip Robinson, Abhinav Gupta, and Kristen Grauman. Audio-visual floorplan reconstruction. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1183–1192, 2021.
[6] Changan Chen, Ziad Al-Halah, and Kristen Grauman. Semantic audio-visual navigation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15516–15525, 2021.
[7] Chuang Gan, Yiwei Zhang, Jiajun Wu, Boqing Gong, and Joshua B Tenenbaum. Look, listen, and act: Towards audio-visual embodied navigation. In 2020 IEEE International Conference on Robotics and Automation (ICRA), pages 9701–9707. IEEE, 2020.
[8] Chuang Gan, Yi Gu, Siyuan Zhou, Jeremy Schwartz, Seth Alter, James Traer, Dan Gutfreund, Joshua B Tenenbaum, Josh H McDermott, and Antonio Torralba. Finding fallen objects via asynchronous audio-visual integration. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10523–10533, 2022.
[9] Daniel G Bobrow. Qualitative reasoning about physical systems: an introduction. Artificial intelligence, 24(1-3):1-5, 1984.
[10] Kenneth D Forbus. Qualitative process theory. Artificial intelligence, 24(1-3):85–168, 1984.
[11] Susan J Hespos, Alissa L Ferry, Erin M Anderson, Emily N Hollenbeck, and Lance J Rips. Five-month-old infants have general knowledge of how nonsolid substances behave and interact. Psychological Science, 27(2):244-256, 2016.
[12] Joan Bliss. Commonsense reasoning about the physical world. Studies in Science Education, 44(2):123-155, 2008.
[13] Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, et al. Piqa: Reasoning about physical commonsense in natural language. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pages 7432-7439, 2020.
[14] Maxwell Forbes, Ari Holtzman, and Yejin Choi. Do neural language representations learn physical commonsense? arXiv preprint arXiv:1908.02899, 2019.
[47] Li Yingzhen and Stephan Mandt. Disentangled sequential autoencoder. In International Conference on Machine Learning, pages 5670-5679. PMLR, 2018.
[48] Prannay Khosla, Piotr Teterwak, Chen Wang, Aaron Sarna, Yonglong Tian, Phillip Isola, Aaron Maschinot, Ce Liu, and Dilip Krishnan. Supervised contrastive learning. Advances in neural information processing systems, 33:18661-18673, 2020.
[49] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference on machine learning, pages 1597-1607. PMLR, 2020.
[50] Shuwen Liu, Bernardo Grau, Ian Horrocks, and Egor Kostylev. Indigo: Gnn-based inductive knowledge graph completion using pair-wise encoding. Advances in Neural Information Processing Systems, 34:2034-2045, 2021.
[51] Yang Ding, Jing Yu, Bang Liu, Yue Hu, Mingxin Cui, and Qi Wu. Mukea: Multimodal knowledge extraction and accumulation for knowledge-based visual question answering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5089-5098, 2022.
[52] Madelyn Glymour, Judea Pearl, and Nicholas P Jewell. Causal inference in statistics: A primer. John Wiley & Sons, 2016.
[53] Judea Pearl and Dana Mackenzie. The book of why: the new science of cause and effect. Basic books, 2018.
[54] Judea Pearl. Direct and indirect effects. In Probabilistic and causal inference: the works of Judea Pearl, pages 373-392. 2022.
[55] Yagya Raj Pandeya and Joonwhoan Lee. Deep learning-based late fusion of multimodal information for emotion classification of music video. Multimedia Tools and Applications, 80: 2887-2905, 2021.
[56] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748-8763. PMLR, 2021.
[57] Andrey Guzhov, Federico Raue, Jorn Hees, and Andreas Dengel. Audioclip: Extending clip to image, text and audio. In ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 976-980. IEEE, 2022.
[58] Yen-Chun Chen, Linjie Li, Licheng Yu, Ahmed El Kholy, Faisal Ahmed, Zhe Gan, Yu Cheng, and Jingjing Liu. Uniter: Universal image-text representation learning. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XXX, pages 104-120. Springer, 2020.
[59] Alane Suhr, Stephanie Zhou, Ally Zhang, Iris Zhang, Huajun Bai, and Yoav Artzi. A corpus for reasoning about natural language grounded in photographs. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 6418-6428, 2019.
[60] Rowan Zellers, Jiasen Lu, Ximing Lu, Youngjae Yu, Yanpeng Zhao, Mohammadreza Salehi, Aditya Kusupati, Jack Hessel, Ali Farhadi, and Yejin Choi. Merlot reserve: Neural script knowledge through vision and language and sound. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16375-16387, 2022.