Disentangled Counterfactual Learning for Physical Audiovisual Commonsense Reasoning
Authors: Changsheng Lv
Shuai Zhang
Yapeng Tian
Mengshi Qi
Huadong Ma
Venue: NeurIPS 2023 poster (2023)
Keywords: Physical Audiovisual；Commonsense Reasoning
TL;DR: We propose a Disentangled Counterfactual Learning (DCL) approach for physical audiovisual commonsense reasoning.

Abstract: In this paper, we propose a Disentangled Counterfactual Learning (DCL) approach for physical audiovisual commonsense reasoning. The task aims to infer objects’ physics commonsense based on both video and audio input, with the main challenge is how to imitate the reasoning ability of humans. Most of the current methods fail to take full advantage of different characteristics in multi-modal data, and lacking causal reasoning ability in models impedes the progress of implicit physical knowledge inferring. To address these issues, our proposed DCL method decouples videos into static (time-invariant) and dynamic (time-varying) factors in the latent space by the disentangled sequential encoder, which adopts a variational autoencoder (VAE) to maximize the mutual information with a contrastive loss function. Furthermore, we introduce a counterfactual learning module to augment the model’s reasoning ability by modeling physical knowledge relationships among different objects under counterfactual intervention. Our proposed method is a plug-and-play module that can be incorporated into any baseline. In experiments, we show that our proposed method improves baseline methods and achieves state-of-the-art performance. Our source code is available at https://github.com/Andy20178/DCL.

Disentangled Counterfactual Learning for Physical Audiovisual Commonsense Reasoning
Changsheng Ly Shuai Zhang4.2,, Yapeng Tian', Mengshi Qil:2et, and Huadong Ma
Beijjig Key Laboratory of Intelligent Telecommunications Software and Multimedia
"Beijig University of Posts and Telecommunications Department of Computer Science, The University of Texas at Dallas (lvchangsheng, zshuai, qms, mhd]@bupt..ed en, yapeng. tian@utaallas edu
Abstract
In this paper, we propose Disentangled Counterfactual Learning (DCL) approach for physical audiovisual commonsense reasoning. The task aims to infer objects' physics commonsense based on both video and audio input, with the main challenge is how to imitate the reasoning ability of humans. Most the current methods fail take full advantage of different characteristics in multi-modal data, and lack- ing causal reasoning ability in models impedes the progress of implicit physical knowledge inferring. To address these issues, our proposed DCL method decouples videos into static (time-invariant) and dynamic (time-varying) factors in the latent space by the disentangled sequential encoder, which adopts variational autoen- coder (VAE) to maximize the mutual information with a contrastive loss function. Furthermore, we introduce counterfactual learning module to augment the model's reasoning ability by modeling physical knowledge relationships among different objects under counterfactual intervention. Our proposed method is plug-and-pe -lay module that can be incorporated into any baseline. In experiments, we show that our proposed method improves baseline methods and achieves sfate-oftthe-eee performance. Our source code is available at htttps//gitabb.com///2/00011
Introduction
Humans acquire physical commonsense knowledge through integrating multi-modal information and have the ability to reason the physical properties of previously unseen objects in the real world [[]
Whether inferring the physical properties of unseen objects ("this object is probably made of wood")) or solving practical problems ('which object would make a bigger mess if dropped") (2) such reasoning abilities represent a sophisticated challenge in the realm of machine intelligence, and play the important role robust robot navigation audiovisual question answering as proxy task to advance the machine's capacity for physical commonsense reasoning.
[ as well as AR/VR [4] this work, we utilize
The major challenge in physical commonsense reasoning is how to develop methods that can extract and reason the implicit physical knowledge from abundant multi-modal data, especially video inputs. This requires the ability to process complex video information, identify the category and the corresponding physical property of each object, and understand the causal relationships among objects. These cognitive processes are similar to those that humans use to know, learn, and reason about the physical world. Current existing methods 6 7 often extract generic visual features from videos that contain human-object interactions, resulting in mixed feature representation that
"Equal contribution
i Corresponning author.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

does not distinguish between object and action information. However, in physical commonsense reasoning, is critical to clearly figure out the attributes and physical properties of objects.
To mitigate the challenge, our idea is to meticulously disentangle the whole video content into static parts and dynamic ones, which means they remain constant features over time, and the features vary with time, respectively. Another driving force behind our work is to establish relationships of physical knowledge among various objects within both video and audio modalities. We enhance the optimization of the results by taking into account the relevance of multiple samples and incorporating causal learning, by applying such relationships as confounders. The application of counterfactual interventions further bolsters the model's explainability capacity.
In this paper, we propose a new Disentangled Counterfactual Learning (DCL) framework for physical audiovisual commonsense reasoning, which explicitly extracts static and dynamic features from video and uses causal learning to mine implicit physical knowledge relationships among various objects. Specifically, we design a Disentangled Sequential Encoder (DSE), which is based on a sequential variational autoencoder clearly separate the static and dynamic factors of the input videos through self-supervision. Meanwhile, we incorporate the contrastive estimation method to enhance the mutual information (MI) between the input data and two latent factors, while also discouraging the MI between the static and dynamic factors. Furthermore, we propose a novel Counterfactual Learning Module (CLM) to capture physical knowledge relationships from a wide array of data samples and employ counterfactual intervention with causal effect tools during the reasoning process. We further refine the training objectives of the model by maximizing the probability likelihood in DSE and the Total Indirect Effect value in CL.. Extensive experimental results on PACS dataset (2) validate that our proposed DCI, approach achieves superior results over previous state-of-ther -h-the--rt art methods, demonstrating our model can effectively improve the reasoning ability and enhance the explainability.
Our main contributions can be summarized as follows:
(I) We propose a new Disentangled Counterfactual Learning (DCL) approach for physical audio- visual commonsense reasoning, which disentangles video inputs into static and dynamic factors via a Sequential Variational Autoencoder. To the best of our knowledge, we are the first to adopt disentanglement learning into this specific task.
(2) We introduce a novel Counterfactual Learning Module to model physical knowledge relationships among various objects, and then apply them as counterfactual interventions to enhance the ability of causal reasoning.
(3) Experimental results demonstrate our proposed DCL is a plug-and-play module that can be incorporated into any baseline, and show DCL can achieve 3.2% absolute improvement over the state-of-the-art methods.
Related Work
Physical Commonsense Reasoning. Naive and intuitive physics was first studied in human psychol- ogy experiments 9 TO 11.. Further research has also been conducted in the multi-sensory perception of physical properties 12] Nowadays, visual and language adopted to learn and utilize physical commonsense in different tasks [13 [4] 15 16 [7]. which only focus on single- modaliyy knowledge and disregard multi-modaa fusion. In embodied audio-visual AI tasks, existing research utilizes multi-modal information to address inverse physics problems 5 6 g]. Different from them, we introduce a disentangled counterfactual learning approach reason the physical commonsense from both video and audio inputs.
Disentangled Representation Learning. Disentangled representation learning aims to learn various hidden explanatory factors behind observable data [8]. which has already been widely applied in computer vision and natural language processing [19
21 22 23 such as image-to-image translation [9]. image generation [20] and text generation 21 21]. However, such methods are limited in the Visual Question Answering (VQA) task. In this paper, propose VAE-based disentangled learning method that disentangles visual information into static and dynamic aspects and improves the interpretability of the model.
20
Causal learning. Traditional VQA datasets suffer from "language prior" 24 or "visual priming bias" (25), leading current methods focus on features' inherent bias of either language or vision and

---

Audio 2
Video
Audio Encoder
Visual Encoder
Objectl
Objectt
Question:
Disentangled Sequence Encoder
Which object would take up you dropped it the ground? Objectl
pick
Counterfactual Learning Module
(b) Input objects and question
P
Question Encoder
F2
4B Kir)
Sampling
Fused Model
Classifier
Fused Model
LSTM
Z
Dy namic distribution g(sb),))
fx.
Ax
Lau
Decoder
R
[F : Fr
YIE
Sampling
distribation
Counterfactual Learning Module
(c) Disentangled Sequence Encoder
X
Disentangled Sequence Encoder
Audio Encoder
Visual Encoder
+1
Audio
Video 1
Counterfactual Relation Interveation
(a) Overview our method
(d) Counterfactual Learning Module
The illustration of our proposed DCL, model: (a) shows the overall architecture: the input videos with audios (b) first encoded by the corresponding visual and audio encoders, then the Disentangled Sequence Encoder (c) is used to decouple video features into static and dynamic factors by LSTM-based VAE. Through the Counterfactual Learning Module (d), we construct the affinity matrix A as a confounder and obtain the prediction Yx.Ax and counterfactual result Yx.Ax- Finally, we achieve YTTE by subtracting these two results and optimizing the model by cross-entropy loss.
Figure 1:
then output inaccurate answers. In recent years, counterfactual thinking and causal reasoning have 28 29], scene graph generation 30
been widely applied in visual explanations 26
27
31
32
33] 34
video understanding [35 36 37 38) 39 and image recognition [31]. Recently, there has been some
work on audio-visual video question answering 40 41 42), but they generally focus cross-modal modeling. Our approach, in contrast, focuses on constructing physical knowledge relationships among different samples and using them confounders in causal reasoning.
3 Proposed Approach
3.1 Overview
Figure[[al illustrates the overall framework of our proposed DCL. First, the model extracts features of input videos and the corresponding audio by visual and audio encoders, respectively. Then, newly designed Disentangled Sequence Encoder (Sec. (3.2p used to decouple the video into static and dynamic factors. Subsequently, the raw and intervened multi-modal features are obtained through Counterfactual Learning Module (Sec. [33], (.3c, and then fused with the question feature. Finally, the prediction result is obtained by optimizing the classification results based on the raw and intervened features (Sec. SSc).
Problem Formulation. As shown in Figure given pair of videos (i.e., < U1, U2 >) with the corresponding audios (i.e., < 01, 02 >). where each video contains an object, and our task is to select more appropriate object from the video inputs to answer the questions (i.e., about the physical commonsense. We define the estimated answer as Y and the ground-truth answer as Y in the dataset. During the pre-processing, we denoted the extracted features of audio, video, and question text X X", and X', respectively, where X X" x
Rd d denotes the feature dimension. Especially,

---

we capture the audio feature and text feature as non-seqeence while extracting the video feature as sequence data X" x2 ;Xy], where T refers the number of video frames.
3.2 Disentangled Sequential Encoder
The proposed Disentangled Sequential Encoder, as shown in Figure [c), can separate the static and dynamic factors for the multi-modal data (we take the video data the example in the follow- ing), which extends the traditional sequential variational autoencoder by incorporating the mutual information term.
Specifically, we assume the latent representations of the input video's feature Xi,r can divide into the static factor and the dynamic factors 21.T, where is the latent dynamic representation at time step Following 43 we assume such two factors are independent of each other, i.e., p(8, ) p(s p((=))(=() where p(-) represents the probability distribution. And 2j depends on Fei = 21, .-: 2;-1). where 2g and the reconstruction of x. is independent of other frames conditioned 2; and s. Hence we need to learn posterior distribution q s/21.7) where the two factors are disentangled formulated as:
(12IT,//E11) = - ( (((url( q( (() II (Z(lE<n)z)))
(1)
We model the posterior distribution by Bi-LSTM [5], where g Zgi) is conditioned on the entire time series by taking the hidden states the input, and q(s æ1.T) can be computed by input T1.T. Then we sample the two decoupled factors s and E1.T by utilizing the distribution ((8).1rr) and q(z:l=ei, xgg)) via the reparameterization trick 46 Afterward, we use the obtained two disentangled factors to reconstruct the Tı.r based on VAE-based decoder [44). The prior of static factor 8 and dynamic factor z, can be defined as the Gaussian distribution with NO,, I) and N (u(zzz), o² (z<x)). respectively, where ul-) and (-)) are modeled by Bi-LSTM. The reconstruction process can be formalized by the factorization:
T
p( 8, p(s) II
Furthermore, we introduce the mutual information to encourage the disentangled two factors mutually exclusive and incorporate non-parnnn con-parameter contrastive estimation into the standard loss function for learning latent representations
which can be formulated as:
o 21.TT xir) d(E1T, Σラ= ((.T,,
C(E1T) = Epo log
(3)
where t means the video has the same object as "positive example", while x², 21, n.. is set of n "negative" videos that have different objects. In order to alleviate the high dimensionality curse 49 we use o (111T, ( = (ep(sim((211)) //Tr)//),, where sim(:.-) means cosine similarity function and r = 0.5 is temperature parameter. C(s) can be calculated in the similar way.
Following 44
content augmentation by randomly shuffling the time steps of the video, and performing motion augmentation via the Gaussian blur [49. The results can be denoted as C(2l))) and C(,T)), where z/ir and " are the augmented data of #1.7 and s, respectively. The MI term can be formulated as follows:
MI MI(ZIT;:11)) x (((zr)) + C(HTr))
MI(s;ZIT) 2 (C(s))+(())) C(s")).
(4)
The objective function can be formulated by adding MI terms to the vanilla ELBO as follows:
T
LDSE = By(emmde
log logp(æ./s 21) + (KTTg]
T
Σ KTlg(z/ass)llo/aa/zel
-a - M.3 + +0 MT(zir;;)) s),
(5)
For the sake of simplicity, in the following content, will use z; to represent X..

---

where a, B and 0 are
The full proof can be found in 44
3.3 Counterfactual Learning Module
In this module, we obtain the decoupled static and dynamic factors s and 21:T after performing the Disentangled Sequential Encoder. Then, these factors with the corresponding audio features sent to construct physical knowledge relationships. Meanwhile, we utilize the counterfactual relation intervention to improve knowledge learning.
Inspired by Knowledge Graph 50 51, the physical knowledge contained in different samples may have certain correlation. Therefore, we propose to model such implicit relationships as graph structure and draw the affinity matrix to represent this physical knowledge relationship among various objects. Similarly, build affinity matrix for audio features or other modal data. With the well-constructed affinity matrix A, we can enhance the static, dynamic, and audio features denoted X,, X and X through passing the message and transferring across various samples, formulated as:
Physical Knowledge Relationship.
X x=
x X X
Ax = Ax" Axy
Ax: |, F Fx
(6)
where Ax represents the augmented matrix composed of three affinity matrices, while F refers to the transferred features, and denotes the transpose of matrix. To achieve A, we take Axy as an example: first compute the similarity matrix based on the static features and filter out the noisy relationships by defining a near neighbor chosen function as T(: k) to retain the top-k values in each row, and then adopt the Laplacian matrix D of S' to normalize the total affinities as the following:
sim(ai, Fzg)
ghe = exp(
Dz E X,
='=
Ax, = D
(7)
where sim is the cosine similarity. The calculation of A", and A" is similar to A
Counterfactual Relation Intervention. To add additional supervision of the affinities Ax, present to highlight the role of the object's physical knowledge relationship during the optimization. First, we express our method as a Structural Causal Model (SCM) [52 53 as shown Figure dd), and then introduce causal inference into our method. Ý represents the final classification output of the model, which is obtained by feeding forward F imput into the fusion model and classifier:
Yx.Ax CCS CLS((Fssion(FF F2,
(8)
where Fj and F2 denote the fused feature of input video pair in and V2, and X denotes the feature of the question text. 'CLS' and "Fusion' represent the classifier and fusion model, respectively. Obviously, the process of deriving the output Y from input X be regarded as types of effects: one is the direct effect X Y. and the other is an indirect type X + Ax + Y. Our final loss function is to maximum likelihood estimation, which will have an end-to-end effect on both types of effects, resulting in insufficient enhancement of A in the indirect effects path, Thus we leverage the Total Indirect Effect (TIE) (54) here to underline the Ax:
YTIE = 8x,Ax - Ex- (X [x.
99)
where Yx refers calculating the results by replacing the original affinity Ax a intervened one Ax-, and X is the given intervened inputs. Note that Yx.Ax- cannot happen in the real world because features X and affinities Ax- come from different X and X which is called counterfactual intervention. Therefore, we modify Yx.Ax to Yx.Ay. is equal to keeping all features fixed but only changing the affinity A, which can highlight the critical effect of A. We compute the expectation of that effect to get the more stable one, and the intervened input features X* are sampled by Gaussian distribution:
X==
X,
X
(10)
where W is the standard random vector whose dimension as same as X, and the mean X,, and standard deviation X,, are both learned by the re- pa-aamttterizaiionn trick in the end-to-end learning.
5

---

3.4 Fusion Model and Optimization
As we have stated, our approach is plug-and-play module that can be directly integrated into various multimodal fusion methods. We take Late fusion [55] as an example to illustrate the structure of fusion models, as well as the optimization function of our DCL. The fusion model can be expressed in the following formula:
Fusion (Fi, P2, x*) = MLPP(Conaa(MMLPPCCCCOCCC
(11)
where Concal represents row-wise concatenation, and MP represents multi-layer perception. Finally, our final optimization goal can be expressed as:
Liotal = LDSE + LTIE.
(12)
where LDSE denote the optimization function we described in Eq. (5) and LTTE denotes the cross- entropy loss of Xx,Ax in Eq. (9) given the ground-truth Y.
Experiments
4.1
Experimental Setup
Dataset. The Physical Audiovisual CommonSense Reasoning Dataset (PACS) 2 is collection of 13,400 question-ane -enswr pairs designed for testing physical commonsense reasoning capabilities. The dataset contains 1,377 unique physical commonsense questions spanning various physical properties and is accompanied by 1,526 videos and audio clips downloaded from YouTube. The total number of data points in the PACS dataset is 13,400, and for PACS- divide PACS into 11,0 11,044/1, 644 as train/val/test set, each of which contains 1,224/150/152 objects respectively. We partitioned the PACS-Material subset into 3,460/444/445 for train/val/test under the same object distribution as PACS. To ensure fair evaluation of models, we evaluate our method on both the entire dataset and a subset focused on material-related problems, reporting the results separately for each subset during testing.
it is 4,349. Following [2]. We
Evaluation Metric. We follow the prior work 2 and use the accuracy to evaluate both PACS and PACS- material subsets. To evaluate the effectiveness of our static factor via disentanglement learning, we measure the accuracy of the PACS-materts material subset using only the static factor. In our experiments, we report results by running the experiments five times and taking the average following (2].
Implementation. We implement our proposed model with PyTorch on two NVIDIA RTX 3090 GPUs. Specifically, we downsampled each video T frames during pre-processing and set the feature dimension as d = 256. In the Disentangled Sequence Encoder, we used a hidden layer size of 256 for Bi-LSTM. During optimization, we set the batch size as 64, which consisted of 64 video pairs and the corresponding questions. In the Counterfactual Learning Module, T = 2 and k = 5 were used when calculating similarities and constructing the physical knowledge relationships. More details about hyper- parameters analysis can be found in our supplementary.
Compared Baseline Methods. To demonstrate the effectiveness of our method across multiple models, we apply it to the following baselines: 1) Late fusion (55), which adopts independent text, image, audio, and video encoders obtain features, and the unimodal embeddings are con- catenated and passed through a linear layer to create multimodal embeddings for prediction. 2) CLIP/AudioCLIP [56] (5). which embed video, text, and audio data into shared vector space using CLIP and AudioCLIP, and a linear layer is used to create multimodal embeddings for prediction. Notes that CLIP is unable extract audio features, so did not use audio in our experiments with CLIP. 3) UNITER (58), which is a pre-trained image and text model for four different image-texi tasks and performs well on tasks such as NLVR2 [59]. As for all of the bbove-mmentioned baseline methods, we adopt the reported parameters following their corresponding paper. Note that another state-of-the-art Merlot Reserve (60) is large-scale multimodal pretraining model that learns joint representations of audio, visual, and language by guiding the new pretraining objectives trained on TPU-v3-512. However, due to the restriction of experimental resources, we are unable to test the
performance of DCL with this model and leave it as future work.

---

Accuracy (%)
Accuracy (%)
Baseline Model
Baseline Model
PACS
Materall
PACS
AACS-Material
Late Fusion 55
55.0 + 1.1
67.4 + 1.5
Late Fusion [55
55.0 1.1
67.4 : 1.5
Late Fusion w/ MLP
54.9 0.9
67.7 + 1.1
Late Fusion w/ DCL
57.7 1 0.9
69.7 i: 1.2
Late Fusion w/ DSE
56.2 0.8
68.5 + 1.2
CLIP [56]
56.3 = 0.7
72.4 1.1
CLIP[56]
56.3 0.7
72.4 + 1.1
CLIP w/ DCL.
58.4 + 0.8
75.4 + 1.2
CLIP w/ ML.
56.5 + 0.5
72.6 + 1.2
AudioCLIP [57
60.0 0.9
75.9 1.1
CLIP w/ DSE
57.0 -1: 0.6
73.2 +: 1.1
AudioCLIP w/ DCL
63.2 = 0.8
76.2 + 1.4
AudioCLIP [5
60.0 + 0.9
75.9 + 1.1
UNITER((argg [58]
60.6 2.2
75.0 2.8
AudioCLIP w/ MLP 60.3 + 0.8
76.2 +: 1.3
UNITER w/ DCL*
62.0 1 2.4
75.7 2.8
AudioCLIP w/ DSE 61.1 + 0.8
76.5 + 1.0
Comparison of our DCL with different baselines.
Ablation study of DSE.
Accuracy (%)
Accuracy (%)
Baseline Model
Baseline Model
PACS
PACS-Material
Matt-Material
Late Fusion 155 w/ DSH
56.2 + 0.8
68.5 + 1.2
Late Pusion 55
w/ dynamic
61).4 + 1.3
Late Fusion w/ DSE,A
57.0 = 1.1
68.9 + 1.3
Late Fusion w/ static
66.0 + 0.6
5.6
Late Fusion w/ DSE,A,C
57.7 = 69.7 + 1.2
Fusion static, dynamic
69.7 1.2
CLIP [56] w/ DSE
57.0 = 0.6
73.2 + 1.1
CLIP [56] w/ dynamic
63.0 = 0.7
CLIP w/ DSE,A
57.8 + 0.8
74.5 + 1.1
CLIP w/ static
71.1 + 0.9
8.1
CLIP w/ DSE,A.C
58.4
75.4 + 1.2
CLIP w/ static, dynamic
75.4 1.2
4.3
AudioCLIP (57 w/ DSE
61.1 = 0.8
76.0 + 1.0
AudioCLIP [56] w/ dynamic
67.1 + 0.6
AudioCLIP w/ DSE,
61.9 = 0.9
75.8 + 1.1
AudioCLIP w/ static
72.5 - 1.0
5,4
AudioCLIP w/ DSE,A,C
63.2 0.8
76.2 + 1.4
AudioCLIP w/ static, dynamic
76.2 1.4
3.7
(c) Ablation study of CL.M
(d) Ablation analysis of the static factors.
Table I: Experimental results and analysis in terms of the accuracy achieved on PACS and PACS- Material. 'DSE' refers to Disentangled Sequential Encoder, while 'CLM' refers to Counterfactual Learning Module. 'A' represents physical knowledge relationship, and 'C' stands for Counterfactual Relation Intervention. A represents the performance improvement compared with the previous
4.2 Comparison to Baselines
Quantitative Results. We report the quantitative performance comparisons using the PACS dataset Table The results show that with the addition of our proposed DCL., both the Late Fusion and UNITER models register absolute improvements of 2.7% and 1.4%, respectively. Similarly, CLIP and AudioCLIP models, which align the image, audio, and text into a shared space, achieve absolute improvements of 2.1% and 3.2%, respectively. These results highlight the strong reasoning and generalization capabilities of our DCL method.
Interestingly, even with the DCL, CLIP's performance trails that of AudioCLIP, emphasizing the critical role of audio information. When comparing CLIP and AudioCLIP enhanced with our DCL, it is evident that the inclusion of audio information leads to an absolute improvement of 4.8%. The same conclusion applies the PACS-Material subset, indicating that our method, serving as plug-and-play module, can consistently enhance material reasoning performance and be flexibly incorporated across various models. noteworthy that all objects in the test set were absent from the training and validation sets, attesting to the zero-shot reasoning ability of our model.
Qualitative Results, order provide a comprehensive qualitative perspective on our results, we present the visualized results of different models for the same questions in Figure 2 In Figure a the two objects exhibit significant visual and auditory differences, facilitating accurate inference by both the baseline model and our DCL. Conversely, the two objects in Figure 2 share similar auditory and static information but differ significantly dynamic information. Our method accurately infers the answers to both questions. This is especially notable in the case of the second question, which requires dynamic information for reasoning, thus highlighting the efficacy of our DCL in the decoupling of static and dynamic elements.
7

---

object!
object2
objectl
object2
objectl
object2
Muterials powder Muterials textiles
Materials
Materials wood
Muterials
Materiuls
Question:
Question
Question:
Which object would be more likely keep egg from cruching when
Which object would more likely flout if dropped water?
If repeutedly hammered both objects, which would be less likely to split in half firs??
dropped from 10 feet?
Properties: flex, hard
Properties, hard, weight
Propertiex: hard, strength
Late fasion:
Objectl
Late fasion:
Objectl
Late fusion
Object
Late fusion w/DCL:
Objectl
Late fasion wDDCL:
Object2
Late fusion w/DCL:
Objectl
A :
Object!
AudioCLIP:
Objectt
AudioCLIP:
Objectt
AudioCLIP WDDC:
Objectl
AudioCLIP w/DC:
Object
AndioCLIP w/DCL:
Object2
Question:
Question:
Question:
Which object would be less likely to shape
Which object could best be used roll out a pie crust a cherey ple?
Which object is more suitable use to dig
was
placed of
ground?
Properties:
shape,
Propertien: hard, strength
weight, texture
Properties: weight
Late fasion:
Object2
Late fusion:
Object2
Late fusion:
Object2
Late fusion w/DCL:
Object1
Late fusion w/DCL.
Object2
Late fusion w/DCL:
Object2
AudioCLIP:
Objectl
AudioCLIP:
Object2
AudioCLIP:
Object2
AudioCLIP w/DCL:
Objectl
AudioCLIP
Object
AudioCLIP wDCL:
Object2
(a)
(b)
(c)
Figure 2: Qualitative Results of bascline w/ and w/o our DCL, where "Material" refers the material of the object, and "Properties" refers to the physical properties that the question focuses on. The correct answers are depicted in green while the incorrect ones are depicted in red.
In Figure C(), only AudioCLIP enhanced with our DCL accurately answers the first question, while the other models fail. This discrepancy arises from inaccurate labeling within the dataset, where two objects appear in the same video, and the bounding box appears to have been assigned to the wrong object. By integrating more robust object detection model and fine-grained annotations, our proposed DCL can circumvent such issues.
4.3 Ablation Study
In this section, we perform the ablation studies to examine the effectiveness of each of our proposed modules, including the Disentangled Sequential Encoder, physical knowledge relationship matrix, and counterfactual relation intervention in the Counterfactual Learning Module.
Disentangled Sequential Encoder (DSE). Table bb) shows the performance comparison of our proposed DSE with different baselines. And we can see AudioCLIP can achieve the best results. After incorporating DSE, all baselines are able to achieve significant improvements in PACS and However, adding MLP module with the same parameters as DSE did not yield such improvement, indicating that the proposed DSE can effectively enhance the ability physical characteristic expression in video features.
Counterfactual Learning Module (CLM). Table (c) presents the ablative analysis of two designed parts in the Counterfactual Learning Module (CLM). After adding the physical knowledge relationship denoted as affinity matrix A, baselines showed an improvement in performance. Similarly, the same result can be clearly found when adding the Counterfactual Relation Intervention denoted as 'C'. Additionally, the same conclusion can drawn from the PACS-Materin demonstrating that both parts of our Counterfactual Learning Module can independently improve results and achieve optimal performance when used simultaneously.
4.4
The Analysis of Disentanglement Learning
In this section, we carry out additional experiments and analysis focusing the decoupled static and dynamic factors obtained through the proposed Disentangled Sequential Encoder. Specifically, we utilize separated individual factors to predict the object's material information, excluding multi-modal

---

Dynamic factor for objects in mini-batch
Similar strength, resilience, flakiness, durability...
Similar in size, weight, material, thickness...
Similar in texture, malleabillity material, distribution...
Figure 3: The t-SNE visualization of the obtained dynamic factors in the latent space of video samples in the test set. The points located close to each other as a cluster exhibit similar or identical actions, such rubbing with the hand (left) and friction with the hand (right), while such two clusters pulled away showing they have completely different dynamic information.
feature fusion. For the captured dynamic factors, we present the t-SNE visualization of various video samples to illustrate the distribution of these dynamic factors within the latent space.
Analysis of static factor. The material classification results are shown in Table [d), where "w/ static" and "w/ dynamic" respectively signify reasoning using only decoupled static factors and dynamic factors, while "w/ static, dynamic" denotes using both static and dynamic factors simultaneously. can be observed that combining the static and dynamic factors significantly improves the accuracy of reasoning for material problems, compared to using only dynamic factors. This indicates the crucial importance of static factors in material property inference. When predicting using only static factors, there is no significant reduction accuracy. confirming the adequacy of static factors in providing sufficient material-relatte information that is not captured by dynamic factors. This also implies that the static and dynamic factors contain little overlapping information and validate the effectiveness of comparative estimation of mutual information (MI).
Analysis of dynamic factor. For dynamic factors, we decoupled the test set videos and showed the t-SNE visualization of the results, as shown in Figure 3 It can be observed that there is dynamic factors. By displaying the
certain clustering phenomenon in the latent representation corresoonding videos of the data points, we found that the points on the left panel of the figure mostly represent actions involving rubbing with objects by hand. this case, the visual features are often insufficient to fully represent the characteristics of the object due to its thickness weight, and the audio information is often the main basis for human reasoning. On the other hand, the points on the right side of the figure correspond to actions involving the friction of objects. Such actions can reveal more dynamic properties of the object, such as malleability. In this case, the dynamic information in the video is often the main basis for human reasoning. Through the above analysis, that for different types of dynamic information, humans will also focus on different modalities of information. This is helpful for future improvements of reasoning models.
5 Conclusion
In this paper, we introduced a method named Disentangled Counterfactual Learning (DCL) for physical audiovisual commonsense reasoning, in which Disentangled Sequential Encoder was designed to decouple the video into static and dynamic factors represented as inne-invriiatt and time- varied information related to actions, respectively. Additionally, we model the physical knowledge relationship among different target objects as an affinity matrix and apply counterfactual relation intervention to guide the model to emphasize the physical commonalities among different objects. Designed as a plug-and-play component, our method can be readily incorporated and has demonstrated its potential to significantly enhance multiple baselines.

---

Acknowledgement
This work was partly supported by the Funds for Creative Research Groups of China under Grant 61921003, the National Natural Science Foundation of China under Grant 62202063, Beijing Natural Science Foundation (No.L223002). the Young Elite Scientists Sponsorship Program by China As- sociation for Science and Technology (CAST) under Grant 2021 I QNRCCO01, the 1 Project under Grant B18008, the Open Project Program of State Key Laboratory of Virtual Reality Technology and Systems, Beihang University (No. VRLAB2022C01).
References
[1] Nikolaus Kriegeskorte. Deep neural networks: a new framework for modeling biological vision and brain information processing. Annual review of vision science, 1:417-446, 2015.
[2] Samuel Yu, Peter Wu, Paul Pu Liang, Ruslan Salakhutdinov, and Louis-Philippe Morency. Pacs:
A dataset for physical audiovisual commonsense reasoning. In Computer Vision-ECCV 2022:
1 77th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XXXVII, pages 292-309. Springer, 2022.
[3] Christopher J Conti, Aparna Varde, and Weitian Wang- Robot action planning by monsense knowledge in human-robot collaborative tasks. In 2020 IEEE International 10T, Electronics and Mechatronics Conference (IEMTRONICS) pages 1-7. IEEE, 2020.
[4] Jean-Luc Lugrin and Mare Cavazza. Making sense of virtual environments: action represen- tation, grounding and common sense. Proceedings of the 12th international conference on Intelligent user interfaces, pages 225-234- 20077.
[5] Senthil Purushwalkam, Sebastia Vicenc Amengual Gari, Vamsi Krishna Ithapu, Carl Schissler, Philip Robinson, Abhinav Gupta, and Kristen Grauman. Audio-visual floorplan reconstruction. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 1183-
1192, 2021.
[6] Changan Chen, Ziad Al-Halah, and Kristen Grauman. Semantic audio-visual navigation. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 15516-15525, 2021.
[7] Chuang Gan, Yiwei Zhang, Jiajun Wu, Boqing Gong, and Joshua T'nnenbaum Look, listen, and act: Towards audio-visual embodied navigation. In 2020 IEEE International Conference on Robotics and Automation (ICRA), pages 9701-9707. IEEE, 2020.
[8] Chuang Gan, Yi Gu, Siyuan Zhou, Jeremy Schwartz, Seth Alter, James Traer, Dan Gutfreund. Joshua B Tenenbaum, Josh H McDermott, and Antonio Torralba. Finding fallen objects via asynchronous audio-visual integration. In Proceedings of the IEEF/CV Conference on Computer Vision and Pattern Recognition, pages 10523-10533, 2022.
[9] Daniel G Bobrow. Qualitative reasoning about physical systems: an introduction. Artificial intelligence, 24(1-3):1-5, 1984.
[10] Kenneth D Forbus. Qualitative process theory. Artificial intelligence, 24(1-3):85-168. 1984.
[11] Susan J Hespos, Alissa I. Ferry, Erin M Anderson, Emily N Hollenbeck, and Lance Rips. Five-month-old infants have general knowledge of how nonsolid substances behave and interact. Psychological Science, 27(2):244-256, 2016.
[12] Joan Bliss. Commonsense reasoning about the physical world. Studies in Science Education. 14(2):123-1555 2008
[13] Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, et al. Piqa: Reasoning about phys- ical commonsense in natural language. In Proceedings of the AAAI conference artificial intelligence, volume 34, pages 7432-7439, 2020.
[14] Maxwell Forbes, Ari Holtzman, and Yejin Choi. Do neural language representations learn physical commonsense? arXiv preprint arXiv: 1908. 19080099,00,999, 2019.
10

---

[15] Yixin Chen, Siyuan Huang, Tao Yuan, Siyuan Qi, Yixin Zhu, and Song. Chun Zhu. Holis- tic++ scene understanding: Single-view 3d holistic scene parsing and human pose estimation with human-object interaction and physical commonsense. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pages 8648 000888867,111 2019.
[16] Yong-Lu Li, Xinpeng Liu, Xiaoqian Wu, Yizhuo Li. Zuoyu Qiu, Liang Xu, Yue Xu, Hao-Shu Fang, and Cewu Lu. Hake: knowledge engine foundation for human activity understanding. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022.
[17] Teresa Wilcox, Rebecca Woods, Lisa Tuggy, and Roman Napoli. Shake. rattle, and... one or two objects? young infants' use of auditory information individuate objects. Infancy, 91):
97-123, 2006.
[18] Yoshua Bengio, Aaron Courville, and Pascal Vincent. Representation learning: A review and new perspectives. IEEE transactions on pattern analysis and machine intelligence, 35(8): 1798-1828, 2013.
[19] Hsin-Ying Lee, Hung-Yu Tseng, Jia-Bin Huang, Maneesh Singh, and Ming-Hsuan Yang. Diverse image-to-image translation via disentangled representations. In Proceedings of the European conference on computer vision (ECCV), pages 35-51, 51, 2018.
[20] Jun-Yan Zhu, Zhoutong Zhang, Chengkai Zhang, Jiajun Wu, Antonio Torralba, Josh Tenen- baum, and Bill Freeman. Visual object networks: Image generation with disentangled 3d representations. Advances in neural information processing systems, 31, 2018.
[21] Ruidan He, Wee Sun Lee, Hwee Tou Ng, and Daniel Dahlmeier. An unsupervised neural attention model for aspect extraction. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Pupers), pages 388-397, 2017.
[22] Yu Bao, Hao Zhou, Shujian Huang, Lei Li, Lili Mou, Olga Vechtomova, Xin-yu Dai, and Jiajun Chen. Generating sentences from disentangled syntactic and semantic spaces. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguisties, pages 6008-601 Florence, Italy, July 2019. Association for Computational Linguistics. doi: 1111666/11/11916660 URL /tttss://aclanttot http://ccaacooooooooor////////// org//19-11111
[23] Hong Chen, Yudong Chen, Xin Wang, Ruobing Xie, Rui Wang, Feng Xia, and Wenwu Zhu. Curriculum disentangled recommendation with noisy multi-feedback. Advances in Neural Information Processing Systems, 34:26924-26936, 2021.
[24] Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh. Making the v in vqa matter: Elevating the role of image understanding in visual question answering. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 6904 6913, 2017.
[25] Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C Lawrence Zitnick, and Devi Parikh. Vqa: Visual question answering. In Proceedings of the IEEE international conference computer vision, pages
2015.
[26] Yash Goyal, Ziyan Wu, Jan Ernst, Dhruv Batra, Devi Parikh, and Stefan Lee. Counterfactual visual explanations. In International Conference on Machine Learning, pages 2376-2384. PMLR, 2019.
[27] Lisa Anne Hendricks, Ronghang Hu, Trevor Darrell, and Zeynep Akata. Grounding visual explanations. In Proceedings of the European conference computer vision (ECCV), pages 264-279, 2018
[28) Pei Wang and Nuno Vasconcelos. Scout: Self-aware discriminant counterfactual explanations. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8981-8990, 2020.
[29] Kexin Yi, Chuang Gan, Yunzhu Li, Pushmeet Kohli, Jiajun Wa, Antonio Torralba, and Joshua B. Tenenbaum. CLEVRER: collision events for video representation and reasoning. ICLR, 2020.
11

---

[30] Kaihua Tang, Yulei Niu, Jianqiang Huang, Jiaxin Shi, and Hanwang Zhang. Unbiased scene graph generation from biased training. In Proceedings of the IEEE/CVE conference on computer vision and pattern recognition, pages 3716-3725, 2020.
[31] Long Chen, Hanwang Zhang, Jun Xiao, Xiangnan He, Shiliang Pu, and Shih-Fu Chang. Counterfactual critic multi-agent training for scene graph generation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 4613-4623, 2019.
[32] Mengshi Qi. Weijian Li, Zhengyuan Yang, Yunhong Wang, and Jiebo Luo. Attentive relational networks for mapping images to scene graphs. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 3957-3966, 2019.
[33] Mengshi Qi, Yunhong Wang, Jie Qin, and Annan Li. Ke-gan: Knowledge embedded generative adversarial networks for semi-superrised scene parsing. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5237-5246, 2019.
[34] Mengshi Qi, Yunhong Wang, Annan Li, and Jiebo Luo. Ste-gan: Spatio- temporally coupled generative adversarial networks for predictive scene parsing. IEEE Transactions on Image Processing, 29:5420-5430, 2020.
[35] Mengshi Qi, Jie Qin, Xiantong Zhen, Di Huang, Yang, and Jiebo Luo. Few-shot ensemble learning for video classification with slowfast memory networks. In Proceedings of the 28th ACM International Conference on Multimedia, pages 3007-3015, 2020.
[36] Mengshi Qi, Yunhong Wang, Annan Li, and Jiebo Luo. Sports video captioning via attentive motion representation and group relationship modeling. IEEE Transactions on Circuits and Systems for Video Technology, 30(8):2617-2633 2019.
[37] Mengshi Qi, Jie Qin, Yi Yang, Yunhong Wang, and Jiebo Luo. Semantics-aware spatial- temporal binaries for cross cross-modal modal video retrieval. IEEE Transactions on Image Processing, 30: 2989-3004, 2021.
[38] Mengshi Qi, Jie Qin, Wu, and Yi Yang. Imitative non-autoreerressvve modeling for trajectory forecasting and imputation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 12736-12745, 2020.
[39] Mengshi Qi, Jie Qin, Annan Li, Yunhong Wang, Jiebo Luo, and Luc Van Gool. stagnet: An attentive semantic mn for group activity recognition. In Proceedings of the European conference on computer vision (ECCV), pages 101-117, 2018.
[40] Heeseung Yun, Youngjae Yu, Wonsuk Yang, Kangil Lee, and Gunhee Kim. Pano-avqa: Grounded audio-visual question answering on 360deg videos. In Proceedings of the IEFF/CVF International Conference Computer Vision, pages 2031-2041, 2021.
[41] Guangyao Li, Yake Wei, Yapeng Tian, Chenliang Xu, Ji-Rong Wen, and Di Hu. Learning to answer questions in dynamic audio-visual scenarios. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 19108-19118, 2022.
[42] Guangyao Li, Wenxuan Hou, and Di Hu. Progressive spatio-temporal perception for audio- visual question answering. arXiv preprint arXiv:2308.00 05421, 2023.
[43] Yizhe Zhu, Martin Renqiang Min, Asim Kadav, and Hans Peter Graf. S3vae: Self-supervised sequential vae for representation disentanglement and data generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6538-668 1338-6547,44 2020.
[44] Junwen Bai, Weiran Wang, and Carla P Gomes. Contrastively disentangled sequential variational autoencoder. Advances in Neural Information Processing Systems, 14:10105-10118, 2021.
[45] Alex Graves and Jürgen Schmidhuber. Framewise phoneme classification with bidirectional Istm and other neural network architectures. Neural networks, 18(5-6):622-610 2005.
[46] Diederik P Kingma and Max Welling. Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114, 2013.
12

---

[47] Li Yingzhen and Stephan Mandt. Disentangled sequential autoencoder. In International Conference on Machine Learning, pages 5670-5679. PMLR, 2018.
[48] Prannay Khosla, Piotr Teterwak, Chen Wang, Aaron Sarna, Yonglong Tian, Phillip Isola, Aaron Maschinot, Ce Liu, and Dilip Krishnan. Supervised contrastive learning. Advances in neural information processing systems, 33: 13:18661-1867, 2020.
[49] Ting Chen, Simon Korblithh, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference machine learning, pages 1597-1607. PMLR, 2020.
[50] Shuwen Liu, Bernardo Grau, Ian Horrocks, and Egor Kostylev. Indigo: Gnn-based inductive knowledge graph completion using pair-wise encoding. Advances in Neural Information Processing Systems, 34:2034 2045, 2021.
[51] Yang Ding, Jing Yu, Bang Liu, Yue Hu, Mingxin Cui, and Qi Wu. Mukea: Multimodal knowledge extraction and accumulation for knowledge-based visual question answering. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 5089-5098, 2022.
[52] Madelyn Glymour, Judea Pearl, and Nicholas P Jewell. Causal inference in statistics: A primer. John Wiley & Sons, 2016
[53] Judea Pearl and Dana Mackenzie. The book of why: the new science of cause and effect. Basic books, 2018.
[54] Judea Pearl. Direct and indirect effects. In Probabilistic and causal inference: the works of Judea Pearl, pages 373-392. 2022.
[55] Yagya Raj Pandeya and Joonwhoan Lee. Deep learning-based late fusion of multimodal information for emotion classification of music video. Multimedia Tools and Applications, 80: 2887-2905, 2021.
[56] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning. pages 8748-8763. PMLR, 2021.
[57] Andrey Guzhov, Federico Raue, Jörn Hees, and Andreas Dengel. Audioclip: Extending clip to image, text and audio. In ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 776-980. IEEE, 2022.
[58] Yen- Yen-Chun Chen, Linjie Li, Licheng Yu, Ahmed El Kholy, Faisal Ahmed, Zhe Gan, Yu Cheng, and Jingjing Liu. Uniter: Universal image-text representation learning. In Computer Vision- ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XXX, pages 104-120. Springer, 2020.
[59] Alane Suhr, Stephanic Zhou, Ally Zhang, Iris Zhang, Huajun Bai, and Yoav Artzi A corpus for reasoning about natural language grounded in photographs. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 6418-6428, 2019.
[60] Rowan Zellers, Jiasen Lu, Ximing Lu, Youngjae Yu, Yanpeng Zhao, Mohammadreza Salehi, Aditya Kusupati, Jack Hessel, Ali Farhadi, and Yejin Choi. Meriot reserve: Neural script knowledge through vision and language and sound. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16375-16387, 2022.
13


Reviews:
[{"review_id": "VyDZQIC2f0", "reviewer": "Reviewer_ZZdS", "summary": "This paper proposes a disentangled counterfactual learning (DCL) approach to solve physical audio-visual commonsense reasoning. This approach first decouples videos into static and dynamic latent features, and then uses a causal learning module to augment the model's reasoning ability. Authors show that this module can be used to augment any existing baselines.", "questions": "My main question about the paper is regarding the performance margin and I'd appreciate authors' clarification on that.", "limitations": "The limitation of this work has not been discussed but the societal impact has been discussed in the last paragraph.", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "The proposed approach is novel and inspiring for similar reasoning tasks. The approach section has detailed each sub-module of the approach. I also like the fact that this approach can be plugged into any existing work to improve its performance.\n\nThe qualitative examples in Fig. 2 show how baseline models augmented with the proposed approach performs better than those without.", "weaknesses": "The performance improvement against baselines is pretty marginal, especially it did not run Merlot Reserve with the additional module. I understand that it was because the computational resource is restricted. But still, since Merlot Reserve has a 10% improvement over the second-best baseline, it remains a question whether this proposed module still brings benefits to Merlot Reserve.\n\nSome other comments: \n\n1. Random performance (guessing by chance) should be shown in Table 1.\n2. The reported baselines are baselines in the benchmark but that does not represent the current SOTA in this domain anymore. There have been many other works coming out after CLIP for example.\n3. Late Fusion w/ dynamic [42] should be written as Late Fusion [42] w/ dynamic. And the same applies to other rows in Table 1.\n4. You need a more concrete example to motivate the physical knowledge relationship module. It is unclear in the paper what kind of correlation between different samples are shared.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Response to Reviewer ZZdS\nThank you for your time and valuable comments, below is our response to your review:\n## Q1: Marginal Performance of our method.\nMerlot Reserve utilizes a large-scale private dataset (YT-Temporal-1B) and a specialized device (v3-1024 TPU) for training. Due to this limitation, we were unable to reimplement the experimental setup of Merlot Reserve.\nTo address concerns about the performance gap, we evaluated the model ERNIE-ViL [1] as a replacement for Merlot Reserve. This is because ERNIE-ViL and Merlot Reserve (base) achieved comparable performance on the VCR task [3]. Additionally, we incorporated the SOTA model BLIP [2] after CLIP in our experiments. The results are as follows:\n| Model| PACS | Relative Improvement (%) |\n|----------------------|:-------------:|:-------------------------:|\n| Random| 50.4|-|\n| Late Fusion|55.0 ± 1.1|-|\n| Late Fusion w/DCL|57.7 ± 0.9| 4.9|\n| CLIP| 56.3 ± 0.7|-|\n| CLIP w/ DCL| 58.4 ± 0.8| 3.7|\n| AudioCLIP  | 60.0 ± 0.9  | -    |\n| AudioCLIP w/ DCL  | 63.2 ± 0.8  | 5.3  |\n| BLIP [2]   | 59.1 ± 0.5  | -    |\n| BLIP w/ DCL   | 61.5 ± 0.6  | 4.0  |\n| UNITER (Large)   | 60.6 ± 2.2  | -   |\n| UNITER (Large) w/DCL | 62.0 ± 2.4  | 2.3   |\n| ERNIE-ViL [1]    | 66.7 ± 1.1  | -    |\n| ERNIE-ViL w/ DCL | 70.4 ± 0.8  | 5.5  |\n\nAs shown in the table, ERNIE-ViL outperforms the second-best baseline by 6%, and with our DCL module, it achieves an improvement of 5.5%. Although it cannot directly prove the effectiveness of our module on Merlot Reserve, based on the results of the six tested models mentioned above, it is reasonable to speculate that our method is still applicable to Merlot Reserve. \n## Q2: Random performance, and typo in Table 1. \nThanks for the suggestion! In the revised version, we will fix these typos and add the random performance in Table 1, denoted as \"Random,\" with a result of 50.4%.\n## Q3: Other works coming out after CLIP.\nWe appreciate your valuable suggestion. We conducted experiments using more powerful baselines BLIP[2] and ERNIE-ViL [1]. The experimental results are shown in the Table in Q1, after incorporating our module, we achieved 4.0% and 5.5% improvement, respectively, confirming the efficacy of our plug-and-play mudole. Based on the results of the six tested models mentioned above, we believe that our module can enhance the model's audio-visual physical commonsense reasoning ability. We will include the additional results in the modified paper.\n## Q4: A more concrete example of the motivation of physical knowledge relationship.\nWe take a piece of data from PACS as an example:\n\nQuestion: Which object could be shattered or bent if the other was to strike it forcefully?\n\nObject 1: A large, thick glass jar with a glassy appearance, making a glass-like sound when struck.\n\nObject 2: A small-sized metal shell cellphone, thin in profile with a metallic sheen, producing a metallic sound upon friction.\n\nOur model reasoning for such an example using the following approach:\n\n1. We aim for the physical knowledge relationships to serve as guidance for the similarity between objects. Through the module, even though object2 may not exhibit the attribute of 'hardness', the shared attribute of hardness among other metallic objects should influence the model's reasoning. For zero-shot objects in the test set (in PACS, all objects in the test set are zero-shot), our approach can leverage attributes from a broader range of objects to aid in reasoning.\n\n2. For both dynamic and static factors and audio information, we employ the near-neighbor chosen function to extract the Top-5 highest similarity scores from each row, resulting in the affinity matrix A. By utilizing Eq.(6), we derive the final features. Our investigation reveals that in the similarity matrix for dynamic factors, object 1's Top-5 exclusively involve videos depicting striking actions on the object, while object 2's Top-5 encompass actions involving friction. In the static factor, object 1's Top-5 include 2 objects characterized by large volumes and 3 ones possessing a glassy texture, while object 2's Top-5 consist of small metallic sheets. Concerning audio, our exploration demonstrates that object 1's Top-5 emit glass-like sounds, while object 2's Top-5 produce sounds akin to sanding. In the process of reasoning for the question, the key lies in identifying object 1's attributes of 'large volume', and 'fragility', whereas object 2 possesses a 'small volume' and 'hardness'. Therefore, the final reasoning yields the result that object 1 is the answer. Our method implicitly represents these features and identifies objects similar to object 1 and object 2 in dynamic, static, and audio aspects, aiding the model in reasoning about the given problem.\n\n3. The reasoning process of our proposed method aligns with the findings of Piloto's work [4], which states that human cognition involves the continuous induction of physical concepts, where specific physical concepts are learned through the similarities in physical knowledge across diverse objects. In the PACS dataset, each question is linked to its corresponding physical concept. These concepts encompass both static attributes such as 'color,' 'thickness', as well as dynamic attributes like 'stretchiness', and 'softness'. Within these physical concepts, objects exhibit distinct attributes. Objects sharing similar physical attributes often yield analogous answers to a given question. Our method capitalizes on this characteristic to assist the model in reasoning.\n\n[1] Ernie-vil: Knowledge enhanced vision-language representations through scene graphs. AAAI 2021\n\n[2] Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation. ICML 2022\n\n[3] From Recognition to Cognition: Visual Commonsense Reasoning. CVPR 2018\n\n[4] Intuitive physics learning in a deep-learning model inspired by developmental psychology. Nature human behaviour 2022"}, {"review_id": "G5gpXfFIDO", "reviewer": "Reviewer_QHKe", "summary": "The paper introduces a novel approach for physical audiovisual commonsense reasoning by Disentangled Counterfactual Learning (DCL). The authors propose a disentangled sequential VAE to separate static and dynamic factors in the visual latent space with an additional contrastive loss term. In addition, a causal learning module that leverages counterfactual intervention between different objects is used to enhance the learning of physical knowledge relations. The proposed modules could be easily plugged into existing baselines. The experiments on PACS dataset demonstrate that the proposed method could improve baseline methods.", "questions": "1. In the current formulation, only the video features are represented as sequence data, but the audio feature is not. What is the reason for doing that? Based on the results, AudioCLIP actually works better than CLIP. This means probably audio plays a more important role in the reasoning. I guess having a disentangled sequential encoder for audio features and using contrastive loss could be possible.\n\n2. Following the first question, the disentangled sequential encoder is for unimodal only (visual). I wonder whether adding a cross-modal mutual information term would be helpful if both audio and visual features are considered sequential.\n\n3. As I mentioned in the weakness, the final performance is still around 60% accuracy, even with the proposed method. While I see some examples in qualitative results and analyses, like inaccurate labeling, I would like to ask what the main failure cases would be and the challenges to overcome.\n\n4. Currently, the method is end-to-end training. I wonder whether it would be helpful if the disentangled sequence VAE is pre-trained first so that the static and dynamic factors are well-learned before applying counterfactual intervention.", "limitations": "N/A", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The idea of modeling implicit physics knowledge relationships between various objects from audio-visual data is interesting and well-motivated.\n\n2. The proposed method is a general module that could be plugged into any baseline. I could also see these modules are definitely not limited to being applied for audio-visual commonsense reasoning.\n\n3. The paper is well-written and straightforward to understand.\n\n4. Extensive experiments and analyses have been done to demonstrate the contribution of each proposed component.", "weaknesses": "1. While I like the general idea of the proposed modules, the final performance shows a minor increment (1.4-3.2%) compared to the baselines. With all these sophisticated designs of additional components, I would expect a larger gap in terms of performance, even though I believe audiovisual physical commonsense reasoning is a challenging task. If I understand correctly, the questions in PACS are binary (e.g., object 1 or 2). As I look into the results, the accuracy is about 60% which is not much better than a simple random guess. In the Supplementary material, there is a model size comparison between models with/without DCL. With the DCL, about 12M more parameters are introduced. Is it possible that the improvement is simply from these additional parameters? A simple verification would be adding the same amount of parameters via MLPs on top of the existing baselines and reporting the outcomes without using DCL.\n\n2. In Section 3.2, Line 137, the authors mention the full proof could be found in Appendix, but it is missing.\n\n3. The T-SNE plots in Figure 3 and the Supplementary material could be more informative. The description of each color in the plot is missing.\n\n4. See Questions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Response to Reviewer QHKe\nThank you for your time and valuable comments, below is our response to your comments:\n## Q1: The impact of  more parameters.\nThank you for your appreciation of our model. Audiovisual physical commonsense reasoning is indeed a formidable challenge (we will elaborate on the specific difficulties shortly).\n\nWe added four layers of MLP with an equivalent number of parameters (12M) to replace our DCL into AudioCLIP, yielding the following results:\n| Model| PACS |\n|--|:--:|\n| Audio CLIP| 60.0 ± 0.9|\n| AudioCLIP w/ MLP| 59.9 ± 0.6|\n| AudioCLIP w/ DCL| 63.2 ± 0.8|\n\nThe results clearly indicate that, even with comparable parameter settings, AudioCLIP w/ MLP remains unable to match the performance of AudioCLIP w/ DCL, affirming the effectiveness of our designed DCL module.\n\n## Q2: Missing proof.\nThe proofs presented in Section 3.2 are standard for sequential VAEs [1,2], so we omitted them for brevity. Please kindly see Appendix A in [1] for the full proofs. We will incorporate these proofs into the appendix of our final version.\n## Q3: More information of the t-SNE plots in Figure.3 and the Supplementary.\nThanks for the suggestion! We have included more comprehensive t-SNE visualizations in our rebuttal material (see Figure 1). \n## Q4: The experiment of disentangled sequential encoder for audio features and using contrastive loss.\nGreat suggestion! Following the baselines in PACS [3], we have abstained from treating audio as sequential data for fair comparison.\n \nHowever, we indeed apply a disentangled sequential encoder to audio features and results are shown in Table 2 of our Supplementary. Decoupling the audio led to a relative improvement of 2.5% in the model's performance. We will incorporating the results into the main paper.\n\nDuring training of the disentangled sequential encoder, we utilized contrastive loss, as outlined in Eq.(3)(4)(5) in the main paper, and the results also demonstrate its effectiveness.\n## Q5: An experiment of adding a cross-modal mutual information term.\nThanks for suggestions. Per your request, we simultaneously treat both visual and audio data as sequential data and disentangle them concurrently. Subsequently, we introduce cross-modal mutual information between the two modalities. The results are as follows:\n| Model|PACS|PACS-Material|\n|--|:--:|:--:|\n| AudioCLIP| 60.0±0.9|75.9±1.1|\n| Audio CLIP w/ DCL (visual)| 61.5±0.8 |76.0±1.4|\n| Audio CLIP w/ DCL (audio)| 63.2±0.8 |76.2±0.7|\n| Audio CLIP w/ DCL (visual and audio)|66.7±0.6 |79.4±1.1|\n| Audio CLIP w/ DCL (visual and audio) w/ MI| 65.5±0.8|78.6±0.5|\n\nAs shown in the table, it is clear that simultaneous decoupling of both audio and visual modalities leads to superior results. However, we did not observe any additional enhancement when we included the mutual information term. We speculate that this is because visual and audio modalities could contain irrelevant information, and adding cross-modal mutual information could actually be detrimental to the results.\n## Q6: What the main failure cases would be and the challenges to overcome.\nThe challenges of our task can be summarized into three aspects:\n\n1. Visual data contains a significant amount of noises.\n\nWithin videos, there exists an abundance of noises such as diverse video backgrounds, and other irrelevant objects.  Extracting generic features using a simple visual encoder yields unsatisfactory results.\n\nMain failure case: In the main paper's Fig.1, in object1's video, besides the tall glass, the screws below serve as irrelevant objects, and the complex background of object2's video introduces noises.\n \n2. The cause of audio generation is difficult to reason.\n\nWithin audio, it is challenging to determine the relationship between actions and sounds. The same object, when interacted with different actions, can produce different sounds, thereby affecting the reasoning of its attributes. Extracting generic features using a single-modal audio encoder yields unsatisfactory results.\n\n3. The questions' setting is challenging.\n\nFor example, in the VQA2.0 [4] and VCR [5] dataset, questions such as \"What color are her eyes?\" necessitate a single round of reasoning to yield an answer. In contrast, within the PACS dataset and audiovisual commonsense reasoning, questions like \"Which object would be less likely to retain its shape if the other was placed on top of it?\" mandate the simultaneous comprehension of the physical attributes of two objects. This intricate configuration of multi-dimensional, multi-step reasoning compounds the difficulty. The binary labels in the dataset is insufficient to represent the real Ground Truth.\n\nMain failure case: In main paper Fig3 (c), the binary labels erroneously guide the model to believe that \"objects that sound rough are not suitable for digging holes\", whereas the actual ground truth is that larger objects are not suitable for digging small holes.\n## Q7: Pre-trained VAE training for our method\nThanks! We conducted additional experiments following your suggestion, and the results are as follows:\n| Model | PACS | PACS-Material |\n|--|:--:|:---:|\n| CLIP w/ DCL -End to End| 56.3±0.7 | 72.4±1.1|\n| CLIP w/ DCL -Pre-trained VAE | 55.2±0.3 | 70.1±1.5|\n\nwhere 'CLIP w/ DCL -Pre-trained VAE' is the method of first training VAE and then train other models, while the other one is our original end-to-end method. As shown in the table, the pre-trained VAE's performance is inferior to that of the end-to-end approach.\n\n[1]S3vae: Self-supervised sequential vae for representation disentanglement and data generation. CVPR2020\n\n[2]Contrastively disentangled sequential variational autoencoder. NIPS2021\n\n[3] PACS: A Dataset for Physical Audiovisual CommonSense Reasoning. ECCV2022\n\n[4] Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answering. CVPR2017\n\n[5] From Recognition to Cognition: Visual Commonsense Reasoning. CVPR2018\n\n"}, {"review_id": "jAIJZXgNEY", "reviewer": "Reviewer_j4kk", "summary": "This paper proposes a novel Disentangled Counterfactual Learning (DCL) method for physical audiovisual commonsense reasoning. DCL consists of two main modules: Disentangled Sequential Encoder and Counterfactual Learning Module(CLM). Disentangled Sequential Encoder decouples videos into static (time-invariant) and dynamic (time-varying) factors in the latent space. The causal learning module augments the model’s reasoning ability by modeling physical knowledge relationships among different objects under counterfactual intervention. The experiments show that DCL can be flexibly integrated into baseline methods and improve their performance.", "questions": "Please see weaknesses.", "limitations": "The authors present the broader impact in this paper.", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 1, "contribution": 2, "strengths": "+ The designed DCL is a plug-and-play module that can be incorporated into baseline methods.\n+ The proposed method reaches SOTA performance. The authors also conduct ablation studies to illustrate the effectiveness of each component in their method.", "weaknesses": "- This paper is poor-written. Some expressions are not consistent, e.g., \"causal learning module\" in the abstract and \"counterfactual learning module\" in the introduction.\n- Why causal learning can help commonsense reasoning? Which concrete problem in audiovisual commonsense reasoning can causal learning address? Although the authors introduce causal learning to augment the model's reasoning ability, no deep analyses are provided in this paper. The authors can provide an example or toy experiments to illustrate their motivation.\n- Figure 3 only shows the t-SNE visualization of dynamic factors. Can the authors show t-SNE of static factors and the comparison with baselines? \n-  Why do the authors only use disentangled sequential encoder (DSE) in videos? Can audios be processed by a similar operation?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Response to Reviewer  j4kk\nThank you for your time and valuable comments, below is our response to your review:\n## Q1: Some typos.\nWe will revise the \"causal learning module\" on line 11 of the paper to \"counterfactual learning module\", and make sure that the entire paper is consistent.\n## Q2: Why do we use causal learning?\nThe motivation for using causal learning was primarily inspired by the following three aspects:\n\n1. The binary labels in the dataset are insufficient to represent the real Ground Truth.\n\nWithin the PACS dataset and audiovisual commonsense reasoning, binary labels do not truly correspond to the Ground Truth. For example, different interaction actions also result in distinct audiovisual features, labels are influenced by both the physical attributes (Ground Truth) exhibited by object's audiovisual features and the human-object interaction actions in the video.  In Fig3 (c) of the main paper, object1, when subjected to the same interaction action as object2, produces a similar sound (the sound of metal being rubbed). Addressing these challenges necessitates the utilization of causal learning to enhance the model's reasoning capabilities.\n\n2. Learning from the correlation P(Y|X) can't  identify the causal effect from X to Y\n\nAs mentioned in [1], if we only train the model based on the correlation P(Y|X) without understanding the confounding effect, regardless of the size of the training data, the model will never be able to discern the true causal effect from X to Y. We employ causal learning to guide the model in learning the true causal effect from audiovisual information to physical commonsense.\n\n3. Using intervention to highlight the role of physical knowledge in reasoning\n\nDrawing inspiration from [2], the intervention in confounding factors can shed light on the significance of particular features within the reasoning. We utilize the physical knowledge similarity matrixs of decoupled video and audio features as confounders, through interventions, highlight the role of physical knowledge in reasoning, and makes the physical knowledge similarity matrix more reliable.\n## Q3: Show t-SNE of static factors and the comparison with baseline.\nWe appreciate your valuable suggestion. In our rebuttal material Figure 2 and 3, we have provided a comparison between the t-SNE plots depicting static factors and the t-SNE plots depicting video features of  the baseline (due to the fact that the baseline was not disentangled and only contains video features) . Figure 2 illustrates the t-SNE visualization of video static factors decoupled by AudioCLIP w/ DCL, while Figure 3 showcases the t-SNE visualization of video features by Audio CLIP.\n## Q4: Disentangled sequential encoder (DSE) in audios.\nWe indeed presented the experimental results of audio decoupling in our Supplementary. Please kindly see Supplementary Table 2 and the response to Reviewer QHKe's Q4 and Q5 for more details.\n\n[1] Causality: models, reasoning and inference. Judea Pearl 2000\n\n[2] Counterfactual Intervention Feature Transfer for Visible-Infrared Person Re-identification. ECCV2022\n\n"}, {"review_id": "jYATwOF6iA", "reviewer": "Reviewer_FCQF", "summary": "The work proposes an approach to separate object and action information from videos to improve the model's reasoning capabilities. The proxy-task of audio-visual question answering is used to train the model commonsense concepts of the physical world. Their contribution focuses on learning from introduced counterfactual examples.\nThey aim to maximise mutual information between input data and static and dynamic scene factors while minimizing the mutual information between those two factors.", "questions": "In Figure 1. What is the referenced object? There are screws and a glass. What if there are several objects in the video but the question is ambiguous because it refers to only one object?\n\nExtracting audio as non-sequence seems odd. How long is the audio instance? What about objects that change their sound characteristically? E.g. think about a laundry machine and a tumble dryer. Both may make some sounds which are over a longer time similar but then the laundry machine starts spinning and you can tell them apart. With such sounds it's also hard choose the part of the sound to encode. How is the length choosen? Does it just cover exactly all the video frames? \n\nHow does the approach deal with different lengths of videos? Is this just not regarded?\n\nIt is hard to find an information of how long the video sequence or audio sequence is in general. In the supplementary material, details are given for the implementation of LateFusion and there it can be found that 8 evenly spaced frames are used but does that mean that every single frame is used or are there skipped frames? And what is the framerate?\n\nIn line 175 it should say standard and not stand.\n\nThe object in Figure 2, b) left side is hardly recognizable. It says the material is wood but is this a woodchip or a fake wood coin? It's very hard to see. And what kind of sound does this make? Are those woodchips dropped on a hard surface? It would be good to either choose another object from another image which is more illustrative or to write down somewhere what the reader actually should see.\n\nIn the \"Analysis of dynamic factor\" it is said two times, audio information is often the main basis for human reasoning or dynamic information is often the main basis for human reasoning. I would just leave out those broad claims which can not be answered by training a network. It reads to me like a very general remark without any citation or investigation to back it up and should probably left to the field of HCI.\n\nIn Table 1, how was the error calculated? Were different training runs performed with different random seeds?", "limitations": "The small dataset test set could probably be mentioned or defended somewhere for people who are not right inside the domain. The broader impact section feels like being included just as exercise but could be removed imo. The argument \"if our algorithm is implemented on a robot and malfunctions this could be bad\" is too general to be useful.", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "The paper presents beyond a good intuition also a detailed mathematical definition of all concepts. Both can be followed well. A fair amount of relevant and current baseline methods were used for comparison. The approach seems to be lightweight enough to be trained on one single GPU within reasonable time. \n\nThe ablation studies seem adequate and useful to understand the success of the disentanglement and the performance boost over baselines.", "weaknesses": "Compared with datasets in other fields the used dataset seems just about big enough to test the approach however a test set in the PACS-Material dataset of 152 objects could potentially lack robustness. Other selected 152 objects may change the result by a large margin. However, in this specific domain it is probably hard to generate larger datasets and comparable work uses the same datasets. K-fold cross evaluation could help here to evaluate the result better.\n\nThere is a clear trend that the method performs well in Table 1, however given the error the results could be more impressive. Again, K-fold cross validation with different splits could help.\n\nThe paper mentions a few times that it can be used as a plugin to improve performance on multimodal fusion tasks, however due to the Q&A being an integral part I don't see how this can generalize to e.g. benefit tracking. Due to their claim it would be nice if the authors could illustrate examples of potential tasks that could be improved with this method.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Response to Reviewer  FCQF\nThank you for your time and valuable comments, below is our response to your review:\n## Q1: K-fold cross validation.\nThank you for your suggestion. We conducted the experiments on the PACS-Material subset using the K-fold cross-validation approach you mentioned. The results are as follows:\n| Model                | PACS Material |\n|----------------------|:--------------:|\n| Late Fusion w/ DCL   | 66.2±0.6     |\n| CLIP w/ DCL          | 74.9±0.8     |\n| AudioCLIP w/ DCL     | 75.4±1.0     |\n| UNITER w/ DCL        | 75.2±0.7     |\n## Q2: Potential tasks that could be improved with our method.\nOur proposed method can be extended to other multi-modal scenarios, such as decision-making in the context of autonomous driving. By incorporating our proposed module, the reasoning capability of the decision model can be enhanced. Similarly, in the behavioral reasoning process of intelligent robots, our approach can highlight the role of certain features in the decision-making process, thereby improving interpretability.\n## Q3: Questions about our paper.\nRegarding your questions, we will address them individually:\n\n1. In general, we consider objects that interact with human hands in the video and produce sound to be the referenced objects. In Fig.1, since the glass is held by a person and produces sound when struck, we consider glass to be the referenced object.\n\n2. In the PACS dataset, the average audio length is 7.6 seconds. In PACS, both the objects in the video and the audio are relatively singular, we encoded all the audio corresponding to the videos in our experiments. Considering the temporal pattern of audio data, for cases involving changes in audio, we can treat audio as sequential data for processing in the future. Great suggestions!\n\n3. We uniformly downsampled videos of varying lengths to 252 frames using corresponding proportions.\n\n4. We used the video format of 30 FPS. Based the downsampled 252 frames, we uniformly sampled 8 frames while skipping others. Later, we utilized ViT to extract features from each frame, and then utilized the averaged features as the video representation.\n\n5. We appreciate your pointing out our errors. We will change \"stand\" to \"standard\" on line 175.\n\n6. Thank you for pointing out the issue. We appreciate your suggestion and have adopted it by replacing Fig2 (b) with a new example. The new example is provided in rebuttal material Figure 4.\n\n7. Indeed, as you mentioned, we have identified two relevant works in the field of HCI to support the importance  of the dynamic information [1,2].\n\n8. As you mentioned, the errors in Table 1 are calculated by training with different random seeds and averaging the results to compute the error.\n## Q4: Limitation about our method.\nWe appreciate your valuable suggestion, and we will make modifications to the expression of this part in the revised version.\n\n[1] Enabling Voice-Accompanying Hand-to-Face Gesture Recognition with Cross-Device Sensing. CHI2023\n\n[2] FaceSight: Enabling Hand-to-Face Gesture Interaction on AR Glasses with a Downward-Facing Camera Vision. CHI2021\n"}, {"review_id": "FyfltlZepl", "reviewer": "Reviewer_PAiU", "summary": "This paper presents a groundbreaking Disentangled Counterfactual Learning (DCL) approach for physical audiovisual commonsense reasoning. The main objective of the proposed method is to infer objects' physics commonsense based on both video and audio inputs, effectively mimicking human reasoning abilities. To address the limitations of existing methods in utilizing the diverse characteristics of multimodal data and lacking causal reasoning abilities, the authors introduce the DCL method. Contributions: 1. introducing a novel DCL approach that leverages disentanglement and causal reasoning to improve multimodal data utilization and achieve remarkable performance. 2. propose a novel Counterfactual Learning Module to model physical knowledge relationships. 3. the proposed DCL method is designed as a plug-and-play module, making it adaptable for integration into various baseline models.", "questions": " You can conduct more quanlitative experiments according to the weaknesses.", "limitations": "Limitations have been discussed in the paper", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The paper introduces an innovative framework for tackling the challenging task of physical commonsense reasoning. The authors meticulously design a disentangled sequential encoder and a counterfactual learning module, both of which contribute to the success of the proposed model in addressing this unique visual question answering (VQA) problem. Importantly, the model's modular nature allows for seamless integration with various baseline approaches, thereby enhancing its versatility.\n\nThe experimental evaluation conducted in this study is relatively enough, demonstrating the effectiveness of the proposed method. The results showcase a significant improvement in accuracy on the PACS dataset, validating the model's ability to handle complex reasoning tasks. Moreover, the authors present qualitative results that vividly illustrate how their approach enhances material reasoning performance, further reinforcing the practical relevance and value of their work.\n\nIn terms of the paper's presentation, the English writing is commendable for its clarity and accessibility. The authors effectively convey their ideas, making it easy for readers to grasp the core concepts and understand the technical details without unnecessary complexity.", "weaknesses": "(1) The main experiment is limited to the PACS dataset and a specific physical audiovisual commonsense reasoning task. While the proposed method is presented as a plug-and-play module, it would be valuable to explore its generalizability to other VQA tasks and unseen data distributions. Considering the versatility of baseline models like CLIP, it would be worthwhile to investigate whether this module can be effectively applied in diverse scenarios.\n\n(2) The ablation analysis is not fully convincing. As a plug-and-play module, it is crucial to clarify that the observed improvement is attributed to the unique design of the proposed method rather than an increase in the number of parameters. To strengthen the argument, additional quantitative experiments could be conducted, such as replacing the Disentangled Sequential Encoder (DSE) with a trivial naive module and comparing the results.\n\n(3) The tables presenting the results of the quantitative experiments are not well displayed. To enhance clarity and readability, improvements could be made in the formatting and organization of the tables. Additionally, the t-SNE visualization in Figure 3 could benefit from displaying more distinct clusters and additional samples. The current arrangement of Figure 3 lacks persuasiveness, as the clusters on the 2D space are not significantly distinguishable from each other.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Response to Reviewer  PAiU \nThank you for your time and valuable comments, below are our responses to your concerns:\n## Q1: Applications in diverse scenarios.\nThis is a question worthy of exploration. Our module is equally applicable to other scenarios. Below are our results on the Visual Commonsense Reasoning (VCR) dataset [1].\n| Model   | Q→A  | QA→R | Q→AR |\n|-----------|:------:|:------:|:------:|\n| LateFusion            | 40.2 | 35.6 | 10.1 |\n| LateFusion w/DCL | 45.7 | 40.5 | 20.1 |\n| CLIP                    | 46.5 | 37.4 | 15.3 |\n| CLIP w/ DCL           | 50.0 | 48.5 | 22.5 |\n\nAccording to the results, our method has led to improvements for both Latefusion and CLIP across multiple performance metrics showing the versatility of our proposed method.\n## Q2: Replacing DCL with a trivial naive module.\nThank you for your suggestion. We conducted the suggested experiments by replacing DCL with four layers of MLP . The results are as follows:\n| Model       | PACS |  PACS-Material |\n|--------------------|:----------:|:------------------:|\n| Audio CLIP         | 60.0±0.9 | 75.9±1.1         |\n| AudioCLIP w/ MLP   | 59.9±0.6 | 73.8±0.6         |\n| AudioCLIP w/DCL    | 63.2±0.8 | 76.2±1.4         |\n\n'AudioCLIP w/ MLP' is the result of replacing DCL with MLPs on top of the AudioCLIP. The results clearly indicate that, even with comparable parameter settings, 'AudioCLIP w/ MLP' remains unable to match the performance of 'AudioCLIP w/DCL', affirming the effectiveness of our designed DCL module.\n## Q3: Improvements of Table.1 and a new Fig.3.\nThank you for your suggestion. We will  improve the tables in our paper to improve clarity and readability. To address your concerns about Fig. 3, we will replace it with the new t-SNE plot from the rebuttal materials Figure 1.\n\n[1] From Recognition to Cognition: Visual Commonsense Reasoning CVPR 2018\n"}]

Author rebuttal:
# Our rebuttal material

Meta-review: ive experts reviewed this paper with 4 accepted recommendations and 1 bordering rejection. The area chairs feel that this work makes a very important contribution by introducing a new Physical Audiovisual Commonsense Reasoning dataset. The reviewers did raise some valuable concerns that should be addressed in the final camera-ready version of the paper. The authors are encouraged to make the necessary changes.

It would be also nice if the authors could include some discussions on related work on embodied audio-visual AI tasks (e.g. [1, 2, 3, 4] ) in the final version. These works can also be seen as an inverse physics problem.

[1] Audio-Visual Floorplan Reconstruction. ICCV 2021

[2] Semantic Audio-Visual Navigation. CVPR 2021.

[3] Finding Fallen Objects via Asynchronous Audio-Visual Integration. CVPR 2023

[4] Look, listen, and act: Towards audio-visual embodied navigation. ICRA 2020
Decision: Accept (poster)