Title: Disentangled Counterfactual Learning for Physical Audiovisual Commonsense Reasoning
Venue: NeurIPS 2023
Accepted: yes
Keywords: Physical Audiovisual；Commonsense Reasoning
Abstract: In this paper, we propose a Disentangled Counterfactual Learning (DCL) approach for physical audiovisual commonsense reasoning. The task aims to infer objects’ physics commonsense based on both video and audio input, with the main challenge is how to imitate the reasoning ability of humans. Most of the current methods fail to take full advantage of different characteristics in multi-modal data, and lacking causal reasoning ability in models impedes the progress of implicit physical knowledge inferring. To address these issues, our proposed DCL method decouples videos into static (time-invariant) and dynamic (time-varying) factors in the latent space by the disentangled sequential encoder, which adopts a variational autoencoder (VAE) to maximize the mutual information with a contrastive loss function. Furthermore, we introduce a counterfactual learning module to augment the model’s reasoning ability by modeling physical knowledge relationships among different objects under counterfactual intervention. Our proposed method is a plug-and-play module that can be incorporated into any baseline. In experiments, we show that our proposed method improves baseline methods and achieves state-of-the-art performance. Our source code is available at https://github.com/Andy20178/DCL.

Disentangled Counterfactual Learning for Physical Audiovisual Commonsense Reasoning

Abstract

In this paper, we propose a Disentangled Counterfactual Learning (DCL) approach for physical audiovisual commonsense reasoning. The task aims to infer objects' physics commonsense based on both video and audio input, with the main challenge is how to imitate the reasoning ability of humans. Most of the current methods fail to take full advantage of different characteristics in multi-modal data, and lacking causal reasoning ability in models impedes the progress of implicit physical knowledge inferring. To address these issues, our proposed DCL method decouples videos into static (time-invariant) and dynamic (time-varying) factors in the latent space by the disentangled sequential encoder, which adopts a variational autoencoder (VAE) to maximize the mutual information with a contrastive loss function. Furthermore, we introduce a counterfactual learning module to augment the model's reasoning ability by modeling physical knowledge relationships among different objects under counterfactual intervention. Our proposed method is a plug-and-play module that can be incorporated into any baseline. In experiments, we show that our proposed method improves baseline methods and achieves state-of-the-art performance. Our source code is available at https://github.com/Andy20178/DCL.

1 Introduction

Humans acquire physical commonsense knowledge through integrating multi-modal information and have the ability to reason the physical properties of previously unseen objects in the real world [1].

Whether inferring the physical properties of unseen objects ('this object is probably made of wood') or solving practical problems ('which object would make a bigger mess if dropped') [2], such reasoning abilities represent a sophisticated challenge in the realm of machine intelligence, and play the important role in robust robot navigation [3] as well as AR/VR [4]. In this work, we utilize audiovisual question answering as a proxy task to advance the machine's capacity for physical commonsense reasoning.

The major challenge in physical commonsense reasoning is how to develop methods that can extract and reason the implicit physical knowledge from abundant multi-modal data, especially video inputs. This requires the ability to process complex video information, identify the category and the corresponding physical property of each object, and understand the causal relationships among objects. These cognitive processes are similar to those that humans use to know, learn, and reason about the physical world. Current existing methods [5, 6, 7, 8] often extract generic visual features from videos that contain human-object interactions, resulting in a mixed feature representation that

does not distinguish between object and action information. However, in physical commonsense reasoning, it is critical to clearly figure out the attributes and physical properties of objects.

To mitigate the challenge, our idea is to meticulously disentangle the whole video content into static parts and dynamic ones, which means they remain constant features over time, and the features vary with time, respectively. Another driving force behind our work is to establish relationships of physical knowledge among various objects within both video and audio modalities. We enhance the optimization of the results by taking into account the relevance of multiple samples and incorporating causal learning, by applying such relationships as confounders. The application of counterfactual interventions further bolsters the model's explainability capacity.

In this paper, we propose a new D isentangled C ounterfactual L earning (DCL) framework for physical audiovisual commonsense reasoning, which explicitly extracts static and dynamic features from video and uses causal learning to mine implicit physical knowledge relationships among various objects. Specifically, we design a Disentangled Sequential Encoder (DSE), which is based on a sequential variational autoencoder to clearly separate the static and dynamic factors of the input videos through self-supervision. Meanwhile, we incorporate the contrastive estimation method to enhance the mutual information (MI) between the input data and two latent factors, while also discouraging the MI between the static and dynamic factors. Furthermore, we propose a novel Counterfactual Learning Module (CLM) to capture physical knowledge relationships from a wide array of data samples and employ counterfactual intervention with causal effect tools during the reasoning process. We further refine the training objectives of the model by maximizing the probability likelihood in DSE and the Total Indirect Effect value in CLM. Extensive experimental results on PACS dataset [2] validate that our proposed DCL approach achieves superior results over previous state-of-the-art methods, demonstrating our model can effectively improve the reasoning ability and enhance the explainability.

Our main contributions can be summarized as follows:

(1) We propose a new Disentangled Counterfactual Learning (DCL) approach for physical audiovisual commonsense reasoning, which disentangles video inputs into static and dynamic factors via a Sequential Variational Autoencoder. To the best of our knowledge, we are the first to adopt disentanglement learning into this specific task.

(2) We introduce a novel Counterfactual Learning Module to model physical knowledge relationships among various objects, and then apply them as counterfactual interventions to enhance the ability of causal reasoning.

(3) Experimental results demonstrate our proposed DCL is a plug-and-play module that can be incorporated into any baseline, and show DCL can achieve 3.2% absolute improvement over the state-of-the-art methods.

2 Related Work

Physical Commonsense Reasoning. Naive and intuitive physics was first studied in human psychology experiments [9, 10, 11]. Further research has also been conducted in the multi-sensory perception of physical properties [12]. Nowadays, visual and language are adopted to learn and utilize physical commonsense in different tasks [13, 14, 15, 16, 17], which only focus on single-modality knowledge and disregard multi-modal fusion. In embodied audio-visual AI tasks, existing research utilizes multi-modal information to address inverse physics problems [5, 6, 7, 8]. Different from them, we introduce a disentangled counterfactual learning approach to reason the physical commonsense from both video and audio inputs.

Disentangled Representation Learning. Disentangled representation learning aims to learn various hidden explanatory factors behind observable data [18], which has already been widely applied in computer vision and natural language processing [19, 20, 21, 22, 23], such as image-to-image translation [19], image generation [20] and text generation [21, 21]. However, such methods are limited in the Visual Question Answering (VQA) task. In this paper, we propose a VAE-based disentangled learning method that disentangles visual information into static and dynamic aspects and improves the interpretability of the model.

Causal learning. Traditional VQA datasets suffer from 'language prior' [24] or 'visual priming bias' [25], leading current methods focus on features' inherent bias of either language or vision and

Video 1

Audio 1

Audio Encoder

Counterfactual Learning Module

𝐹

!

∗

Visual Encoder

Disentangled Sequence Encoder

Video 2

Audio 2

Audio Encoder

Counterfactual Learning Module

Visual Encoder

Disentangled Sequence Encoder

𝐹

#

∗

= [𝐹

$

∗

⋮ 𝐹

%

∗

⋮ 𝐹

&

∗

]

Question Encoder

𝐹

!

𝐹

#

= [𝐹

$

⋮ 𝐹

%

⋮ 𝐹

&

]

Classifier

𝑌 '

',)

" ∗

𝑌 '

',)

"

𝑌 '

*+,

(a) Overview of our method

𝑧

!:#

𝑠

𝑋

$

Question:

Which object would take more time to pick

up if you dropped it on the ground?

Answer: Object1

LSTM

𝑞(𝑧

!:#

|𝑥

!:#

)

ℒ

$%

Sampling

Sampling

𝑞(𝑠|𝑥

!:#

)

Static

distribution

Dynamic

distribution

𝑧

!:#

𝑆

Decoder

(c) Disentangled Sequence Encoder

(d) Counterfactual Learning Module

𝑋

𝑌 &

𝐴

-

𝑋

𝑌 &

∗

𝐴

-

∗

𝑋

∗

Counterfactual Relation Intervention

(b) Input objects and question

Object1

Object2

泰

then output inaccurate answers. In recent years, counterfactual thinking and causal reasoning have been widely applied in visual explanations [26, 27, 28, 29], scene graph generation [30, 31, 32, 33, 34], video understanding [35, 36, 37, 38, 39] and image recognition [31]. Recently, there has been some work on audio-visual video question answering [40, 41, 42], but they generally focus on cross-modal modeling. Our approach, in contrast, focuses on constructing physical knowledge relationships among different samples and using them as confounders in causal reasoning.

3 Proposed Approach

3.1 Overview

Figure 1(a) illustrates the overall framework of our proposed DCL. First, the model extracts features of input videos and the corresponding audio by visual and audio encoders, respectively. Then, a newly designed Disentangled Sequence Encoder (Sec. 3.2) is used to decouple the video into static and dynamic factors. Subsequently, the raw and intervened multi-modal features are obtained through the Counterfactual Learning Module (Sec. 3.3), and then fused with the question feature. Finally, the prediction result is obtained by optimizing the classification results based on the raw and intervened features (Sec. 3.4).

Problem Formulation. As shown in Figure 1(b), given a pair of videos ( i.e., < v 1 , v 2 > ) with the corresponding audios ( i.e., < a 1 , a 2 > ), where each video contains an object, and our task is to select a more appropriate object from the video inputs to answer the questions ( i.e., q ) about the physical commonsense. We define the estimated answer as ˆ Y and the ground-truth answer as Y in the dataset. During the pre-processing, we denoted the extracted features of audio, video, and question text as X a , X v , and X t , respectively, where X a , X v , X t ∈ R d , d denotes the feature dimension. Especially,

we capture the audio feature and text feature as non-sequence while extracting the video feature as sequence data as X v = { X v 1 , X v 2 , · · · , X v T } , where T refers to the number of video frames.

3.2 Disentangled Sequential Encoder

The proposed Disentangled Sequential Encoder, as shown in Figure 1(c), can separate the static and dynamic factors for the multi-modal data (we take the video data as the example in the following), which extends the traditional sequential variational autoencoder by incorporating the mutual information term.

Specifically, we assume the latent representations of the input video's feature X v 1: T can divide into the static factor s and the dynamic factors z 1: T , where z t is the latent dynamic representation at time step t . Following [43, 44], we assume such two factors are independent of each other, i.e., p ( s, z 1: T ) = p ( s ) p ( z 1: T ) , where p ( · ) represents the probability distribution. And z i depends on z <i = { z 0 , z 1 , .., z i -1 } , where z 0 = 0 and the reconstruction of x i 3 is independent of other frames conditioned on z i and s . Hence we need to learn a posterior distribution q ( z 1: T , s | x 1: T ) where the two factors are disentangled formulated as:

We model the posterior distribution by Bi-LSTM [45], where q ( z i | z <i , x ≤ i ) is conditioned on the entire time series by taking the hidden states as the input, and q ( s | x 1: T ) can be computed by input x 1: T . Then we sample the two decoupled factors s and z 1: T by utilizing the distribution q ( s | x 1: T ) and q ( z i | z <i , x ≤ i ) via the reparameterization trick [46]. Afterward, we use the obtained two disentangled factors to reconstruct the x 1: T based on a VAE-based decoder [44]. The prior of static factor s and dynamic factor z i can be defined as the Gaussian distribution with N (0 , I ) and N ( µ ( z <i ) , σ 2 ( z <i )) , respectively, where µ ( · ) and σ ( · ) are modeled by Bi-LSTM. The reconstruction process can be formalized by the factorization:

Furthermore, we introduce the mutual information to encourage the disentangled two factors mutually exclusive and incorporate non-parameter contrastive estimation into the standard loss function for learning latent representations [43, 47], which can be formulated as:

where x + means the video has the same object as 'positive example', while x j , j = { 1 , 2 , ..., n } is a set of n 'negative' videos that have different objects. In order to alleviate the high dimensionality curse [48, 49], we use ϕ ( z 1: T , x + 1: T ) = exp ( sim ( z 1: T , x + 1: T ) /τ ) , where sim ( · , · ) means cosine similarity function and τ = 0 . 5 is a temperature parameter. C ( s ) can be calculated in the similar way.

Following [44], we use content augmentation by randomly shuffling the time steps of the video, and performing motion augmentation via the Gaussian blur [49]. The results can be denoted as C ( z m 1: T ) and C ( s m ) , where z m 1: T and s m are the augmented data of z 1: T and s , respectively. The MI term can be formulated as follows:

The objective function can be formulated by adding MI terms to the vanilla ELBO as follows:

where γ , α , β and θ are hyper-parameters. The full proof can be found in [44].

3.3 Counterfactual Learning Module

In this module, we obtain the decoupled static and dynamic factors s and z 1: T after performing the Disentangled Sequential Encoder. Then, these factors with the corresponding audio features are sent to construct physical knowledge relationships. Meanwhile, we utilize the counterfactual relation intervention to improve knowledge learning.

Physical Knowledge Relationship. Inspired by Knowledge Graph [50, 51], the physical knowledge contained in different samples may have a certain correlation. Therefore, we propose to model such implicit relationships as a graph structure and draw the affinity matrix A to represent this physical knowledge relationship among various objects. Similarly, we build an affinity matrix for audio features or other modal data. With the well-constructed affinity matrix A , we can enhance the static, dynamic, and audio features denoted as X v s , X v z , and X a through passing the message and transferring across various samples, formulated as:

where A X represents the augmented matrix composed of three affinity matrices, while F refers to the transferred features, and ⊤ denotes the transpose of a matrix. To achieve A , we take A X v s as an example: we first compute the similarity matrix based on the static features and filter out the noisy relationships by defining a near neighbor chosen function as T ( · , k ) to retain the top-k values in each row, and then adopt the Laplacian matrix D -1 of S ′ to normalize the total affinities as the following:

where sim ( · , · ) is the cosine similarity. The calculation of A v z and A a is similar to A v s .

Counterfactual Relation Intervention. To add additional supervision of the affinities A X , we present to highlight the role of the object's physical knowledge relationship during the optimization. First, we express our method as a Structural Causal Model (SCM) [52, 53] as shown in Figure 1(d), and then introduce causal inference into our method. ˆ Y represents the final classification output of the model, which is obtained by feeding forward F input into the fusion model and classifier:

where F 1 and F 2 denote the fused feature of input video pair v 1 and v 2 , and X t denotes the feature of the question text. ' CLS ' and ' Fusion ' represent the classifier and fusion model, respectively. Obviously, the process of deriving the output ˆ Y from input X can be regarded as two types of effects: one is the direct effect X → ˆ Y , and the other is an indirect type X → A X → ˆ Y . Our final loss function is to maximum likelihood estimation, which will have an end-to-end effect on both types of effects, resulting in insufficient enhancement of A in the indirect effects path. Thus we leverage the Total Indirect Effect (TIE) [54] here to underline the A X :

where ˆ Y X,A X ∗ refers to calculating the results by replacing the original affinity A X to a intervened one A X ∗ , and X ∗ is the given intervened inputs. Note that ˆ Y X,A X ∗ cannot happen in the real world because features X and affinities A X ∗ come from different X and X ∗ , which is called counterfactual intervention. Therefore, we modify ˆ Y X,A X to ˆ Y X,A X ∗ is equal to keeping all features fixed but only changing the affinity A , which can highlight the critical effect of A . We compute the expectation of that effect to get the more stable one, and the intervened input features X ∗ are sampled by a Gaussian distribution:

where W is the standard random vector whose dimension is as same as X , and the mean X µ and standard deviation X σ are both learned by the re-parameterization trick in the end-to-end learning.

3.4 Fusion Model and Optimization

As we have stated, our approach is a plug-and-play module that can be directly integrated into various multimodal fusion methods. We take Late fusion [55] as an example to illustrate the structure of fusion models, as well as the optimization function of our DCL. The fusion model can be expressed in the following formula:

where Concat represents row-wise concatenation, and MLP represents multi-layer perception. Finally, our final optimization goal can be expressed as:

where L DSE denote the optimization function we described in Eq. (5) and L TIE denotes the crossentropy loss of ˆ Y X,A X in Eq. (9) given the ground-truth Y .

4 Experiments

4.1 Experimental Setup

Dataset. The Physical Audiovisual CommonSense Reasoning Dataset (PACS) [2] is a collection of 13,400 question-answer pairs designed for testing physical commonsense reasoning capabilities. The dataset contains 1,377 unique physical commonsense questions spanning various physical properties and is accompanied by 1,526 videos and audio clips downloaded from YouTube. The total number of data points in the PACS dataset is 13,400, and for PACS-Material it is 4,349. Following [2], We divide PACS into 11,044/1,192/1,164 as train/val/test set, each of which contains 1,224/150/152 objects respectively. We partitioned the PACS-Material subset into 3,460/444/445 for train/val/test under the same object distribution as PACS. To ensure a fair evaluation of models, we evaluate our method on both the entire dataset and a subset focused on material-related problems, reporting the results separately for each subset during testing.

Evaluation Metric. We follow the prior work [2] and use the accuracy to evaluate both PACS and PACS-material subsets. To evaluate the effectiveness of our static factor via disentanglement learning, we measure the accuracy of the PACS-material subset using only the static factor. In our experiments, we report results by running the experiments five times and taking the average following [2].

Implementation. We implement our proposed model with PyTorch on two NVIDIA RTX 3090 GPUs. Specifically, we downsampled each video to T = 8 frames during pre-processing and set the feature dimension as d = 256 . In the Disentangled Sequence Encoder, we used a hidden layer size of 256 for Bi-LSTM. During optimization, we set the batch size as 64, which consisted of 64 video pairs and the corresponding questions. In the Counterfactual Learning Module, τ = 2 and k = 5 were used when calculating similarities and constructing the physical knowledge relationships. More details about hyper-parameters analysis can be found in our supplementary.

Compared Baseline Methods. To demonstrate the effectiveness of our method across multiple models, we apply it to the following baselines: 1) Late fusion [55], which adopts independent text, image, audio, and video encoders to obtain features, and the unimodal embeddings are concatenated and passed through a linear layer to create multimodal embeddings for prediction. 2) CLIP/AudioCLIP [56, 57], which embed video, text, and audio data into a shared vector space using CLIP and AudioCLIP, and a linear layer is used to create multimodal embeddings for prediction. Notes that CLIP is unable to extract audio features, so we did not use audio in our experiments with CLIP. 3) UNITER [58], which is a pre-trained image and text model for four different image-text tasks and performs well on tasks such as NLVR2 [59]. As for all of the above-mentioned baseline methods, we adopt the reported parameters following their corresponding paper. Note that another state-of-the-art Merlot Reserve [60] is a large-scale multimodal pretraining model that learns joint representations of audio, visual, and language by guiding the new pretraining objectives trained on TPU-v3-512. However, due to the restriction of experimental resources, we are unable to test the performance of DCL with this model and leave it as future work.

(a) Comparison of our DCL with different baselines.

(c) Ablation study of CLM

(d) Ablation analysis of the static factors.

4.2 Comparison to Baselines

Quantitative Results. We report the quantitative performance comparisons using the PACS dataset in Table 1(a). The results show that with the addition of our proposed DCL, both the Late Fusion and UNITER models register absolute improvements of 2.7% and 1.4%, respectively. Similarly, CLIP and AudioCLIP models, which align the image, audio, and text into a shared space, achieve absolute improvements of 2.1% and 3.2%, respectively. These results highlight the strong reasoning and generalization capabilities of our DCL method.

Interestingly, even with the DCL, CLIP's performance trails that of AudioCLIP, emphasizing the critical role of audio information. When comparing CLIP and AudioCLIP enhanced with our DCL, it is evident that the inclusion of audio information leads to an absolute improvement of 4.8%. The same conclusion applies to the PACS-Material subset, indicating that our method, serving as a plug-and-play module, can consistently enhance material reasoning performance and be flexibly incorporated across various models. It is noteworthy that all objects in the test set were absent from the training and validation sets, attesting to the zero-shot reasoning ability of our model.

Qualitative Results. In order to provide a comprehensive qualitative perspective on our results, we present the visualized results of different models for the same questions in Figure 2. In Figure 2(a), the two objects exhibit significant visual and auditory differences, facilitating accurate inference by both the baseline model and our DCL. Conversely, the two objects in Figure 2(b) share similar auditory and static information but differ significantly in dynamic information. Our method accurately infers the answers to both questions. This is especially notable in the case of the second question, which requires dynamic information for reasoning, thus highlighting the efficacy of our DCL in the decoupling of static and dynamic elements.

Late fusion :

Object1

Late fusion w/DCL :

Object1

AudioCLIP :

Object1

AudioCLIP w/DCL :

Object1

Late fusion :

Object2

Late fusion w/DCL :

Object1

AudioCLIP :

Object1

AudioCLIP w/DCL :

Object1

Question

：

Which object would be more likely to keep an egg from cracking when dropped from 10 feet?

Late fusion :

Object1

Late fusion w/DCL :

Object1

AudioCLIP :

Object1

AudioCLIP w/DCL :

Object1

Question

：

Which object would be more likely to

float if dropped in water?

(a)

object1

object2

object1

object2

Question:

If I repeatedly hammered both objects, which would be less likely to split in half first?

object1

object2

Late fusion :

Object1

Late fusion w/DCL :

Object2

AudioCLIP :

Object1

AudioCLIP w/DCL :

Object2

(b)

Late fusion :

Object1

Late fusion w/DCL :

Object1

AudioCLIP :

Object1

AudioCLIP w/DCL :

Object2

(c)

Question ：

Which object would be less likely to retain it's shape if the other was placed on top of it?

Late fusion :

Object2

Late fusion w/DCL :

Object1

AudioCLIP :

Object1

AudioCLIP w/DCL :

Object1

Question

：

Which object could best be used to roll out a pie crust for a cherry pie?

Question:

Which object is more suitable to use to dig a small hole in the ground?

Late fusion :

Object2

Late fusion w/DCL :

Object2

AudioCLIP :

Object2

AudioCLIP w/DCL :

Object1

Late fusion :

Object2

Late fusion w/DCL :

Object2

AudioCLIP :

Object2

AudioCLIP w/DCL :

Object2

Materials : powderMaterials : textiles

Materials : wood

Materials : wood

Materials : metal

Materials : metal

Properties ： flex, hard

Properties ： hard, weight

Properties ： hard, strength

Properties ： hard, strength

Properties ： hard, shape, weight, texture

Properties ： size, weight

Late fusion :

Object1

Late fusion w/DCL :

Object2

AudioCLIP :

Object1

AudioCLIP w/DCL :

Object2

(b)

Late fusion :

Object2

Late fusion w/DCL :

Object2

AudioCLIP :

Object2

AudioCLIP w/DCL :

Object1

Late fusion :

Object1

Late fusion w/DCL :

Object1

AudioCLIP :

Object1

AudioCLIP w/DCL :

Object2

(c)

Late fusion :

Object2

Late fusion w/DCL :

Object2

AudioCLIP :

Object2

AudioCLIP w/DCL :

Object2

In Figure 2(c), only AudioCLIP enhanced with our DCL accurately answers the first question, while the other models fail. This discrepancy arises from inaccurate labeling within the dataset, where two objects appear in the same video, and the bounding box appears to have been assigned to the wrong object. By integrating a more robust object detection model and fine-grained annotations, our proposed DCL can circumvent such issues.

4.3 Ablation Study

In this section, we perform the ablation studies to examine the effectiveness of each of our proposed modules, including the Disentangled Sequential Encoder, physical knowledge relationship matrix, and counterfactual relation intervention in the Counterfactual Learning Module.

Disentangled Sequential Encoder (DSE). Table 1(b) shows the performance comparison of our proposed DSE with different baselines. And we can see AudioCLIP can achieve the best results. After incorporating DSE, all baselines are able to achieve significant improvements in PACS and PACS-Material. However, adding an MLP module with the same parameters as DSE did not yield such improvement, indicating that the proposed DSE can effectively enhance the ability of physical characteristic expression in video features.

Counterfactual Learning Module (CLM). Table 1(c) presents the ablative analysis of two designed parts in the Counterfactual Learning Module (CLM). After adding the physical knowledge relationship denoted as affinity matrix A , all baselines showed an improvement in performance. Similarly, the same result can be clearly found when adding the Counterfactual Relation Intervention denoted as 'C'. Additionally, the same conclusion can be drawn from the PACS-Material, demonstrating that both parts of our Counterfactual Learning Module can independently improve results and achieve optimal performance when used simultaneously.

4.4 The Analysis of Disentanglement Learning

In this section, we carry out additional experiments and analysis focusing on the decoupled static and dynamic factors obtained through the proposed Disentangled Sequential Encoder. Specifically, we utilize separated individual factors to predict the object's material information, excluding multi-modal

Dynamic factor for objects in a mini-batch

Similarin strength,

resilience,flakiness,

durability...

Similar in size,weight,material,

Similar in texture, malleability,

thickness...

material, distribution...

feature fusion. For the captured dynamic factors, we present the t-SNE visualization of various video samples to illustrate the distribution of these dynamic factors within the latent space.

Analysis of static factor. The material classification results are shown in Table 1(d), where 'w/ static' and 'w/ dynamic' respectively signify reasoning using only decoupled static factors and dynamic factors, while 'w/ static, dynamic' denotes using both static and dynamic factors simultaneously. It can be observed that combining the static and dynamic factors significantly improves the accuracy of reasoning for material problems, compared to using only dynamic factors. This indicates the crucial importance of static factors in material property inference. When predicting using only static factors, there is no significant reduction in accuracy, confirming the adequacy of static factors in providing sufficient material-related information that is not captured by dynamic factors. This also implies that the static and dynamic factors contain little overlapping information and validate the effectiveness of our comparative estimation of mutual information (MI).

Analysis of dynamic factor. For dynamic factors, we decoupled the test set videos and showed the t-SNE visualization of the results, as shown in Figure 3. It can be observed that there is a certain clustering phenomenon in the latent representation of dynamic factors. By displaying the corresponding videos of the data points, we found that the points on the left panel of the figure mostly represent actions involving rubbing with objects by hand. In this case, the visual features are often insufficient to fully represent the characteristics of the object due to its thickness or weight, and the audio information is often the main basis for human reasoning. On the other hand, the points on the right side of the figure correspond to actions involving the friction of objects. Such actions can reveal more dynamic properties of the object, such as malleability. In this case, the dynamic information in the video is often the main basis for human reasoning. Through the above analysis, we can see that for different types of dynamic information, humans will also focus on different modalities of information. This is helpful for future improvements of our reasoning models.

5 Conclusion

In this paper, we introduced a method named Disentangled Counterfactual Learning (DCL) for physical audiovisual commonsense reasoning, in which a Disentangled Sequential Encoder was designed to decouple the video into static and dynamic factors represented as time-invariant and timevaried information related to actions, respectively. Additionally, we model the physical knowledge relationship among different target objects as an affinity matrix and apply counterfactual relation intervention to guide the model to emphasize the physical commonalities among different objects. Designed as a plug-and-play component, our method can be readily incorporated and has demonstrated its potential to significantly enhance multiple baselines.

References

Nikolaus Kriegeskorte. Deep neural networks: a new framework for modeling biological vision and brain information processing. Annual review of vision science , 1:417-446, 2015.

Samuel Yu, Peter Wu, Paul Pu Liang, Ruslan Salakhutdinov, and Louis-Philippe Morency. Pacs: A dataset for physical audiovisual commonsense reasoning. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XXXVII , pages 292-309. Springer, 2022.

Christopher J Conti, Aparna S Varde, and Weitian Wang. Robot action planning by commonsense knowledge in human-robot collaborative tasks. In 2020 IEEE International IOT, Electronics and Mechatronics Conference (IEMTRONICS) , pages 1-7. IEEE, 2020.

Jean-Luc Lugrin and Marc Cavazza. Making sense of virtual environments: action representation, grounding and common sense. In Proceedings of the 12th international conference on Intelligent user interfaces , pages 225-234, 2007.

Senthil Purushwalkam, Sebastia Vicenc Amengual Gari, Vamsi Krishna Ithapu, Carl Schissler, Philip Robinson, Abhinav Gupta, and Kristen Grauman. Audio-visual floorplan reconstruction. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 11831192, 2021.

Changan Chen, Ziad Al-Halah, and Kristen Grauman. Semantic audio-visual navigation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 15516-15525, 2021.

Chuang Gan, Yiwei Zhang, Jiajun Wu, Boqing Gong, and Joshua B Tenenbaum. Look, listen, and act: Towards audio-visual embodied navigation. In 2020 IEEE International Conference on Robotics and Automation (ICRA) , pages 9701-9707. IEEE, 2020.

Chuang Gan, Yi Gu, Siyuan Zhou, Jeremy Schwartz, Seth Alter, James Traer, Dan Gutfreund, Joshua B Tenenbaum, Josh H McDermott, and Antonio Torralba. Finding fallen objects via asynchronous audio-visual integration. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 10523-10533, 2022.

Daniel G Bobrow. Qualitative reasoning about physical systems: an introduction. Artificial intelligence , 24(1-3):1-5, 1984.

Kenneth D Forbus. Qualitative process theory. Artificial intelligence , 24(1-3):85-168, 1984.

Susan J Hespos, Alissa L Ferry, Erin M Anderson, Emily N Hollenbeck, and Lance J Rips. Five-month-old infants have general knowledge of how nonsolid substances behave and interact. Psychological Science , 27(2):244-256, 2016.

Joan Bliss. Commonsense reasoning about the physical world. Studies in Science Education , 44(2):123-155, 2008.

Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, et al. Piqa: Reasoning about physical commonsense in natural language. In Proceedings of the AAAI conference on artificial intelligence , volume 34, pages 7432-7439, 2020.

Maxwell Forbes, Ari Holtzman, and Yejin Choi. Do neural language representations learn physical commonsense? arXiv preprint arXiv:1908.02899 , 2019.

Yixin Chen, Siyuan Huang, Tao Yuan, Siyuan Qi, Yixin Zhu, and Song-Chun Zhu. Holistic++ scene understanding: Single-view 3d holistic scene parsing and human pose estimation with human-object interaction and physical commonsense. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 8648-8657, 2019.

Yong-Lu Li, Xinpeng Liu, Xiaoqian Wu, Yizhuo Li, Zuoyu Qiu, Liang Xu, Yue Xu, Hao-Shu Fang, and Cewu Lu. Hake: a knowledge engine foundation for human activity understanding. IEEE Transactions on Pattern Analysis and Machine Intelligence , 2022.

Teresa Wilcox, Rebecca Woods, Lisa Tuggy, and Roman Napoli. Shake, rattle, and... one or two objects? young infants' use of auditory information to individuate objects. Infancy , 9(1): 97-123, 2006.

Yoshua Bengio, Aaron Courville, and Pascal Vincent. Representation learning: A review and new perspectives. IEEE transactions on pattern analysis and machine intelligence , 35(8): 1798-1828, 2013.

Hsin-Ying Lee, Hung-Yu Tseng, Jia-Bin Huang, Maneesh Singh, and Ming-Hsuan Yang. Diverse image-to-image translation via disentangled representations. In Proceedings of the European conference on computer vision (ECCV) , pages 35-51, 2018.

Jun-Yan Zhu, Zhoutong Zhang, Chengkai Zhang, Jiajun Wu, Antonio Torralba, Josh Tenenbaum, and Bill Freeman. Visual object networks: Image generation with disentangled 3d representations. Advances in neural information processing systems , 31, 2018.

Ruidan He, Wee Sun Lee, Hwee Tou Ng, and Daniel Dahlmeier. An unsupervised neural attention model for aspect extraction. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages 388-397, 2017.

Yu Bao, Hao Zhou, Shujian Huang, Lei Li, Lili Mou, Olga Vechtomova, Xin-yu Dai, and Jiajun Chen. Generating sentences from disentangled syntactic and semantic spaces. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics , pages 6008-6019, Florence, Italy, July 2019. Association for Computational Linguistics. doi: 10.18653/v1/P19-1602. URL https://aclanthology.org/P19-1602 .

Hong Chen, Yudong Chen, Xin Wang, Ruobing Xie, Rui Wang, Feng Xia, and Wenwu Zhu. Curriculum disentangled recommendation with noisy multi-feedback. Advances in Neural Information Processing Systems , 34:26924-26936, 2021.

Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh. Making the v in vqa matter: Elevating the role of image understanding in visual question answering. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 6904-6913, 2017.

Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C Lawrence Zitnick, and Devi Parikh. Vqa: Visual question answering. In Proceedings of the IEEE international conference on computer vision , pages 2425-2433, 2015.

Yash Goyal, Ziyan Wu, Jan Ernst, Dhruv Batra, Devi Parikh, and Stefan Lee. Counterfactual visual explanations. In International Conference on Machine Learning , pages 2376-2384. PMLR, 2019.

Lisa Anne Hendricks, Ronghang Hu, Trevor Darrell, and Zeynep Akata. Grounding visual explanations. In Proceedings of the European conference on computer vision (ECCV) , pages 264-279, 2018.

Pei Wang and Nuno Vasconcelos. Scout: Self-aware discriminant counterfactual explanations. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 8981-8990, 2020.

Kexin Yi, Chuang Gan, Yunzhu Li, Pushmeet Kohli, Jiajun Wu, Antonio Torralba, and Joshua B. Tenenbaum. CLEVRER: collision events for video representation and reasoning. In ICLR , 2020.

Kaihua Tang, Yulei Niu, Jianqiang Huang, Jiaxin Shi, and Hanwang Zhang. Unbiased scene graph generation from biased training. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 3716-3725, 2020.

Long Chen, Hanwang Zhang, Jun Xiao, Xiangnan He, Shiliang Pu, and Shih-Fu Chang. Counterfactual critic multi-agent training for scene graph generation. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 4613-4623, 2019.

Mengshi Qi, Weijian Li, Zhengyuan Yang, Yunhong Wang, and Jiebo Luo. Attentive relational networks for mapping images to scene graphs. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 3957-3966, 2019.

Mengshi Qi, Yunhong Wang, Jie Qin, and Annan Li. Ke-gan: Knowledge embedded generative adversarial networks for semi-supervised scene parsing. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 5237-5246, 2019.

Mengshi Qi, Yunhong Wang, Annan Li, and Jiebo Luo. Stc-gan: Spatio-temporally coupled generative adversarial networks for predictive scene parsing. IEEE Transactions on Image Processing , 29:5420-5430, 2020.

Mengshi Qi, Jie Qin, Xiantong Zhen, Di Huang, Yi Yang, and Jiebo Luo. Few-shot ensemble learning for video classification with slowfast memory networks. In Proceedings of the 28th ACM International Conference on Multimedia , pages 3007-3015, 2020.

Mengshi Qi, Yunhong Wang, Annan Li, and Jiebo Luo. Sports video captioning via attentive motion representation and group relationship modeling. IEEE Transactions on Circuits and Systems for Video Technology , 30(8):2617-2633, 2019.

Mengshi Qi, Jie Qin, Yi Yang, Yunhong Wang, and Jiebo Luo. Semantics-aware spatialtemporal binaries for cross-modal video retrieval. IEEE Transactions on Image Processing , 30: 2989-3004, 2021.

Mengshi Qi, Jie Qin, Yu Wu, and Yi Yang. Imitative non-autoregressive modeling for trajectory forecasting and imputation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 12736-12745, 2020.

Mengshi Qi, Jie Qin, Annan Li, Yunhong Wang, Jiebo Luo, and Luc Van Gool. stagnet: An attentive semantic rnn for group activity recognition. In Proceedings of the European conference on computer vision (ECCV) , pages 101-117, 2018.

Heeseung Yun, Youngjae Yu, Wonsuk Yang, Kangil Lee, and Gunhee Kim. Pano-avqa: Grounded audio-visual question answering on 360deg videos. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 2031-2041, 2021.

Guangyao Li, Yake Wei, Yapeng Tian, Chenliang Xu, Ji-Rong Wen, and Di Hu. Learning to answer questions in dynamic audio-visual scenarios. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 19108-19118, 2022.

Guangyao Li, Wenxuan Hou, and Di Hu. Progressive spatio-temporal perception for audiovisual question answering. arXiv preprint arXiv:2308.05421 , 2023.

Yizhe Zhu, Martin Renqiang Min, Asim Kadav, and Hans Peter Graf. S3vae: Self-supervised sequential vae for representation disentanglement and data generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 6538-6547, 2020.

Junwen Bai, Weiran Wang, and Carla P Gomes. Contrastively disentangled sequential variational autoencoder. Advances in Neural Information Processing Systems , 34:10105-10118, 2021.

Alex Graves and Jürgen Schmidhuber. Framewise phoneme classification with bidirectional lstm and other neural network architectures. Neural networks , 18(5-6):602-610, 2005.

Diederik P Kingma and Max Welling. Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114 , 2013.

Li Yingzhen and Stephan Mandt. Disentangled sequential autoencoder. In International Conference on Machine Learning , pages 5670-5679. PMLR, 2018.

Prannay Khosla, Piotr Teterwak, Chen Wang, Aaron Sarna, Yonglong Tian, Phillip Isola, Aaron Maschinot, Ce Liu, and Dilip Krishnan. Supervised contrastive learning. Advances in neural information processing systems , 33:18661-18673, 2020.

Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference on machine learning , pages 1597-1607. PMLR, 2020.

Shuwen Liu, Bernardo Grau, Ian Horrocks, and Egor Kostylev. Indigo: Gnn-based inductive knowledge graph completion using pair-wise encoding. Advances in Neural Information Processing Systems , 34:2034-2045, 2021.

Yang Ding, Jing Yu, Bang Liu, Yue Hu, Mingxin Cui, and Qi Wu. Mukea: Multimodal knowledge extraction and accumulation for knowledge-based visual question answering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 5089-5098, 2022.

Madelyn Glymour, Judea Pearl, and Nicholas P Jewell. Causal inference in statistics: A primer . John Wiley & Sons, 2016.

Judea Pearl and Dana Mackenzie. The book of why: the new science of cause and effect . Basic books, 2018.

Judea Pearl. Direct and indirect effects. In Probabilistic and causal inference: the works of Judea Pearl , pages 373-392. 2022.

Yagya Raj Pandeya and Joonwhoan Lee. Deep learning-based late fusion of multimodal information for emotion classification of music video. Multimedia Tools and Applications , 80: 2887-2905, 2021.

Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning , pages 8748-8763. PMLR, 2021.

Andrey Guzhov, Federico Raue, Jörn Hees, and Andreas Dengel. Audioclip: Extending clip to image, text and audio. In ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , pages 976-980. IEEE, 2022.

Yen-Chun Chen, Linjie Li, Licheng Yu, Ahmed El Kholy, Faisal Ahmed, Zhe Gan, Yu Cheng, and Jingjing Liu. Uniter: Universal image-text representation learning. In Computer VisionECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XXX , pages 104-120. Springer, 2020.

Alane Suhr, Stephanie Zhou, Ally Zhang, Iris Zhang, Huajun Bai, and Yoav Artzi. A corpus for reasoning about natural language grounded in photographs. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics , pages 6418-6428, 2019.

Rowan Zellers, Jiasen Lu, Ximing Lu, Youngjae Yu, Yanpeng Zhao, Mohammadreza Salehi, Aditya Kusupati, Jack Hessel, Ali Farhadi, and Yejin Choi. Merlot reserve: Neural script knowledge through vision and language and sound. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 16375-16387, 2022.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a groundbreaking Disentangled Counterfactual Learning (DCL) approach for physical audiovisual commonsense reasoning. The main objective of the proposed method is to infer objects' physics commonsense based on both video and audio inputs, effectively mimicking human reasoning abilities. To address the limitations of existing methods in utilizing the diverse characteristics of multimodal data and lacking causal reasoning abilities, the authors introduce the DCL method. Contributions: 1. introducing a novel DCL approach that leverages disentanglement and causal reasoning to improve multimodal data utilization and achieve remarkable performance. 2. propose a novel Counterfactual Learning Module to model physical knowledge relationships. 3. the proposed DCL method is designed as a plug-and-play module, making it adaptable for integration into various baseline models.\n\nSTRENGTHS:\nThe paper introduces an innovative framework for tackling the challenging task of physical commonsense reasoning. The authors meticulously design a disentangled sequential encoder and a counterfactual learning module, both of which contribute to the success of the proposed model in addressing this unique visual question answering (VQA) problem. Importantly, the model's modular nature allows for seamless integration with various baseline approaches, thereby enhancing its versatility.\n\nThe experimental evaluation conducted in this study is relatively enough, demonstrating the effectiveness of the proposed method. The results showcase a significant improvement in accuracy on the PACS dataset, validating the model's ability to handle complex reasoning tasks. Moreover, the authors present qualitative results that vividly illustrate how their approach enhances material reasoning performance, further reinforcing the practical relevance and value of their work.\n\nIn terms of the paper's presentation, the English writing is commendable for its clarity and accessibility. The authors effectively convey their ideas, making it easy for readers to grasp the core concepts and understand the technical details without unnecessary complexity.\n\nWEAKNESSES:\n(1) The main experiment is limited to the PACS dataset and a specific physical audiovisual commonsense reasoning task. While the proposed method is presented as a plug-and-play module, it would be valuable to explore its generalizability to other VQA tasks and unseen data distributions. Considering the versatility of baseline models like CLIP, it would be worthwhile to investigate whether this module can be effectively applied in diverse scenarios.\n\n(2) The ablation analysis is not fully convincing. As a plug-and-play module, it is crucial to clarify that the observed improvement is attributed to the unique design of the proposed method rather than an increase in the number of parameters. To strengthen the argument, additional quantitative experiments could be conducted, such as replacing the Disentangled Sequential Encoder (DSE) with a trivial naive module and comparing the results.\n\n(3) The tables presenting the results of the quantitative experiments are not well displayed. To enhance clarity and readability, improvements could be made in the formatting and organization of the tables. Additionally, the t-SNE visualization in Figure 3 could benefit from displaying more distinct clusters and additional samples. The current arrangement of Figure 3 lacks persuasiveness, as the clusters on the 2D space are not significantly distinguishable from each other.\n\nQUESTIONS:\nYou can conduct more quanlitative experiments according to the weaknesses.\n\nLIMITATIONS:\nLimitations have been discussed in the paper", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe work proposes an approach to separate object and action information from videos to improve the model's reasoning capabilities. The proxy-task of audio-visual question answering is used to train the model commonsense concepts of the physical world. Their contribution focuses on learning from introduced counterfactual examples.\nThey aim to maximise mutual information between input data and static and dynamic scene factors while minimizing the mutual information between those two factors.\n\nSTRENGTHS:\nThe paper presents beyond a good intuition also a detailed mathematical definition of all concepts. Both can be followed well. A fair amount of relevant and current baseline methods were used for comparison. The approach seems to be lightweight enough to be trained on one single GPU within reasonable time. \n\nThe ablation studies seem adequate and useful to understand the success of the disentanglement and the performance boost over baselines.\n\nWEAKNESSES:\nCompared with datasets in other fields the used dataset seems just about big enough to test the approach however a test set in the PACS-Material dataset of 152 objects could potentially lack robustness. Other selected 152 objects may change the result by a large margin. However, in this specific domain it is probably hard to generate larger datasets and comparable work uses the same datasets. K-fold cross evaluation could help here to evaluate the result better.\n\nThere is a clear trend that the method performs well in Table 1, however given the error the results could be more impressive. Again, K-fold cross validation with different splits could help.\n\nThe paper mentions a few times that it can be used as a plugin to improve performance on multimodal fusion tasks, however due to the Q&A being an integral part I don't see how this can generalize to e.g. benefit tracking. Due to their claim it would be nice if the authors could illustrate examples of potential tasks that could be improved with this method.\n\nQUESTIONS:\nIn Figure 1. What is the referenced object? There are screws and a glass. What if there are several objects in the video but the question is ambiguous because it refers to only one object?\n\nExtracting audio as non-sequence seems odd. How long is the audio instance? What about objects that change their sound characteristically? E.g. think about a laundry machine and a tumble dryer. Both may make some sounds which are over a longer time similar but then the laundry machine starts spinning and you can tell them apart. With such sounds it's also hard choose the part of the sound to encode. How is the length choosen? Does it just cover exactly all the video frames? \n\nHow does the approach deal with different lengths of videos? Is this just not regarded?\n\nIt is hard to find an information of how long the video sequence or audio sequence is in general. In the supplementary material, details are given for the implementation of LateFusion and there it can be found that 8 evenly spaced frames are used but does that mean that every single frame is used or are there skipped frames? And what is the framerate?\n\nIn line 175 it should say standard and not stand.\n\nThe object in Figure 2, b) left side is hardly recognizable. It says the material is wood but is this a woodchip or a fake wood coin? It's very hard to see. And what kind of sound does this make? Are those woodchips dropped on a hard surface? It would be good to either choose another object from another image which is more illustrative or to write down somewhere what the reader actually should see.\n\nIn the \"Analysis of dynamic factor\" it is said two times, audio information is often the main basis for human reasoning or dynamic information is often the main basis for human reasoning. I would just leave out those broad claims which can not be answered by training a network. It reads to me like a very general remark without any citation or investigation to back it up and should probably left to the field of HCI.\n\nIn Table 1, how was the error calculated? Were different training runs performed with different random seeds?\n\nLIMITATIONS:\nThe small dataset test set could probably be mentioned or defended somewhere for people who are not right inside the domain. The broader impact section feels like being included just as exercise but could be removed imo. The argument \"if our algorithm is implemented on a robot and malfunctions this could be bad\" is too general to be useful.", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a novel Disentangled Counterfactual Learning (DCL) method for physical audiovisual commonsense reasoning. DCL consists of two main modules: Disentangled Sequential Encoder and Counterfactual Learning Module(CLM). Disentangled Sequential Encoder decouples videos into static (time-invariant) and dynamic (time-varying) factors in the latent space. The causal learning module augments the model’s reasoning ability by modeling physical knowledge relationships among different objects under counterfactual intervention. The experiments show that DCL can be flexibly integrated into baseline methods and improve their performance.\n\nSTRENGTHS:\n+ The designed DCL is a plug-and-play module that can be incorporated into baseline methods.\n+ The proposed method reaches SOTA performance. The authors also conduct ablation studies to illustrate the effectiveness of each component in their method.\n\nWEAKNESSES:\n- This paper is poor-written. Some expressions are not consistent, e.g., \"causal learning module\" in the abstract and \"counterfactual learning module\" in the introduction.\n- Why causal learning can help commonsense reasoning? Which concrete problem in audiovisual commonsense reasoning can causal learning address? Although the authors introduce causal learning to augment the model's reasoning ability, no deep analyses are provided in this paper. The authors can provide an example or toy experiments to illustrate their motivation.\n- Figure 3 only shows the t-SNE visualization of dynamic factors. Can the authors show t-SNE of static factors and the comparison with baselines? \n-  Why do the authors only use disentangled sequential encoder (DSE) in videos? Can audios be processed by a similar operation?\n\nQUESTIONS:\nPlease see weaknesses.\n\nLIMITATIONS:\nThe authors present the broader impact in this paper.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces a novel approach for physical audiovisual commonsense reasoning by Disentangled Counterfactual Learning (DCL). The authors propose a disentangled sequential VAE to separate static and dynamic factors in the visual latent space with an additional contrastive loss term. In addition, a causal learning module that leverages counterfactual intervention between different objects is used to enhance the learning of physical knowledge relations. The proposed modules could be easily plugged into existing baselines. The experiments on PACS dataset demonstrate that the proposed method could improve baseline methods.\n\nSTRENGTHS:\n1. The idea of modeling implicit physics knowledge relationships between various objects from audio-visual data is interesting and well-motivated.\n\n2. The proposed method is a general module that could be plugged into any baseline. I could also see these modules are definitely not limited to being applied for audio-visual commonsense reasoning.\n\n3. The paper is well-written and straightforward to understand.\n\n4. Extensive experiments and analyses have been done to demonstrate the contribution of each proposed component.\n\nWEAKNESSES:\n1. While I like the general idea of the proposed modules, the final performance shows a minor increment (1.4-3.2%) compared to the baselines. With all these sophisticated designs of additional components, I would expect a larger gap in terms of performance, even though I believe audiovisual physical commonsense reasoning is a challenging task. If I understand correctly, the questions in PACS are binary (e.g., object 1 or 2). As I look into the results, the accuracy is about 60% which is not much better than a simple random guess. In the Supplementary material, there is a model size comparison between models with/without DCL. With the DCL, about 12M more parameters are introduced. Is it possible that the improvement is simply from these additional parameters? A simple verification would be adding the same amount of parameters via MLPs on top of the existing baselines and reporting the outcomes without using DCL.\n\n2. In Section 3.2, Line 137, the authors mention the full proof could be found in Appendix, but it is missing.\n\n3. The T-SNE plots in Figure 3 and the Supplementary material could be more informative. The description of each color in the plot is missing.\n\n4. See Questions.\n\nQUESTIONS:\n1. In the current formulation, only the video features are represented as sequence data, but the audio feature is not. What is the reason for doing that? Based on the results, AudioCLIP actually works better than CLIP. This means probably audio plays a more important role in the reasoning. I guess having a disentangled sequential encoder for audio features and using contrastive loss could be possible.\n\n2. Following the first question, the disentangled sequential encoder is for unimodal only (visual). I wonder whether adding a cross-modal mutual information term would be helpful if both audio and visual features are considered sequential.\n\n3. As I mentioned in the weakness, the final performance is still around 60% accuracy, even with the proposed method. While I see some examples in qualitative results and analyses, like inaccurate labeling, I would like to ask what the main failure cases would be and the challenges to overcome.\n\n4. Currently, the method is end-to-end training. I wonder whether it would be helpful if the disentangled sequence VAE is pre-trained first so that the static and dynamic factors are well-learned before applying counterfactual intervention.\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a disentangled counterfactual learning (DCL) approach to solve physical audio-visual commonsense reasoning. This approach first decouples videos into static and dynamic latent features, and then uses a causal learning module to augment the model's reasoning ability. Authors show that this module can be used to augment any existing baselines.\n\nSTRENGTHS:\nThe proposed approach is novel and inspiring for similar reasoning tasks. The approach section has detailed each sub-module of the approach. I also like the fact that this approach can be plugged into any existing work to improve its performance.\n\nThe qualitative examples in Fig. 2 show how baseline models augmented with the proposed approach performs better than those without.\n\nWEAKNESSES:\nThe performance improvement against baselines is pretty marginal, especially it did not run Merlot Reserve with the additional module. I understand that it was because the computational resource is restricted. But still, since Merlot Reserve has a 10% improvement over the second-best baseline, it remains a question whether this proposed module still brings benefits to Merlot Reserve.\n\nSome other comments: \n\n1. Random performance (guessing by chance) should be shown in Table 1.\n2. The reported baselines are baselines in the benchmark but that does not represent the current SOTA in this domain anymore. There have been many other works coming out after CLIP for example.\n3. Late Fusion w/ dynamic [42] should be written as Late Fusion [42] w/ dynamic. And the same applies to other rows in Table 1.\n4. You need a more concrete example to motivate the physical knowledge relationship module. It is unclear in the paper what kind of correlation between different samples are shared.\n\nQUESTIONS:\nMy main question about the paper is regarding the performance margin and I'd appreciate authors' clarification on that.\n\nLIMITATIONS:\nThe limitation of this work has not been discussed but the societal impact has been discussed in the last paragraph.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-13"}