SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Changsheng Lv $^{1,2*}$ , Shuai Zhang $^{1,2*}$ , Yapeng Tian $^{3}$ , Mengshi Qi $^{1,2\boxdot\dagger}$ , and Huadong Ma $^{1,2}$
<sup>1</sup>Beijing Key Laboratory of Intelligent Telecommunications Software and Multimedia
$^{2}$ Beijing University of Posts and Telecommunications
<sup>3</sup>Department of Computer Science, The University of Texas at Dallas
{lvchangsheng, zshuai, qms, mhd}@bupt.edu.cn, yapeng.tian@utdallas.edu
In this paper, we propose a Disentangled Counterfactual Learning (DCL) approach for physical audiovisual commonsense reasoning. The task aims to infer objects' physics commonsense based on both video and audio input, with the main challenge is how to imitate the reasoning ability of humans. Most of the current methods fail to take full advantage of different characteristics in multi-modal data, and lacking causal reasoning ability in models impedes the progress of implicit physical knowledge inferring. To address these issues, our proposed DCL method decouples videos into static (time-invariant) and dynamic (time-varying) factors in the latent space by the disentangled sequential encoder, which adopts a variational autoencoder (VAE) to maximize the mutual information with a contrastive loss function. Furthermore, we introduce a counterfactual learning module to augment the model's reasoning ability by modeling physical knowledge relationships among different objects under counterfactual intervention. Our proposed method is a plug-and-play module that can be incorporated into any baseline. In experiments, we show that our proposed method improves baseline methods and achieves state-of-the-art performance. Our source code is available at https://github.com/Andy20178/DCL.
Humans acquire physical commonsense knowledge through integrating multi-modal information and have the ability to reason the physical properties of previously unseen objects in the real world [1].
Whether inferring the physical properties of unseen objects ("this object is probably made of wood") or solving practical problems ("which object would make a bigger mess if dropped") [2], such reasoning abilities represent a sophisticated challenge in the realm of machine intelligence, and play the important role in robust robot navigation [3] as well as AR/VR [4]. In this work, we utilize audiovisual question answering as a proxy task to advance the machine's capacity for physical commonsense reasoning.
The major challenge in physical commonsense reasoning is how to develop methods that can extract and reason the implicit physical knowledge from abundant multi-modal data, especially video inputs. This requires the ability to process complex video information, identify the category and the corresponding physical property of each object, and understand the causal relationships among objects. These cognitive processes are similar to those that humans use to know, learn, and reason about the physical world. Current existing methods [5, 6, 7, 8] often extract generic visual features from videos that contain human-object interactions, resulting in a mixed feature representation that
does not distinguish between object and action information. However, in physical commonsense reasoning, it is critical to clearly figure out the attributes and physical properties of objects.
To mitigate the challenge, our idea is to meticulously disentangle the whole video content into static parts and dynamic ones, which means they remain constant features over time, and the features vary with time, respectively. Another driving force behind our work is to establish relationships of physical knowledge among various objects within both video and audio modalities. We enhance the optimization of the results by taking into account the relevance of multiple samples and incorporating causal learning, by applying such relationships as confounders. The application of counterfactual interventions further bolsters the model's explainability capacity.
In this paper, we propose a new Disentangled Counterfactual Learning (DCL) framework for physical audiovisual commonsense reasoning, which explicitly extracts static and dynamic features from video and uses causal learning to mine implicit physical knowledge relationships among various objects. Specifically, we design a Disentangled Sequential Encoder (DSE), which is based on a sequential variational autoencoder to clearly separate the static and dynamic factors of the input videos through self-supervision. Meanwhile, we incorporate the contrastive estimation method to enhance the mutual information (MI) between the input data and two latent factors, while also discouraging the MI between the static and dynamic factors. Furthermore, we propose a novel Counterfactual Learning Module (CLM) to capture physical knowledge relationships from a wide array of data samples and employ counterfactual intervention with causal effect tools during the reasoning process. We further refine the training objectives of the model by maximizing the probability likelihood in DSE and the Total Indirect Effect value in CLM. Extensive experimental results on PACS dataset [2] validate that our proposed DCL approach achieves superior results over previous state-of-the-art methods, demonstrating our model can effectively improve the reasoning ability and enhance the explainability.
(1) We propose a new Disentangled Counterfactual Learning (DCL) approach for physical audiovisual commonsense reasoning, which disentangles video inputs into static and dynamic factors via a Sequential Variational Autoencoder. To the best of our knowledge, we are the first to adopt disentanglement learning into this specific task.
(2) We introduce a novel Counterfactual Learning Module to model physical knowledge relationships among various objects, and then apply them as counterfactual interventions to enhance the ability of causal reasoning.
(3) Experimental results demonstrate our proposed DCL is a plug-and-play module that can be incorporated into any baseline, and show DCL can achieve  $3.2\%$  absolute improvement over the state-of-the-art methods.
Physical Commonsense Reasoning. Naive and intuitive physics was first studied in human psychology experiments [9, 10, 11]. Further research has also been conducted in the multi-sensory perception of physical properties [12]. Nowadays, visual and language are adopted to learn and utilize physical commonsense in different tasks [13, 14, 15, 16, 17], which only focus on single-modality knowledge and disregard multi-modal fusion. In embodied audio-visual AI tasks, existing research utilizes multi-modal information to address inverse physics problems [5, 6, 7, 8]. Different from them, we introduce a disentangled counterfactual learning approach to reason the physical commonsense from both video and audio inputs.
Disentangled Representation Learning. Disentangled representation learning aims to learn various hidden explanatory factors behind observable data [18], which has already been widely applied in computer vision and natural language processing [19, 20, 21, 22, 23], such as image-to-image translation [19], image generation [20] and text generation [21, 21]. However, such methods are limited in the Visual Question Answering (VQA) task. In this paper, we propose a VAE-based disentangled learning method that disentangles visual information into static and dynamic aspects and improves the interpretability of the model.
Causal learning. Traditional VQA datasets suffer from "language prior" [24] or "visual priming bias" [25], leading current methods focus on features' inherent bias of either language or vision and
(a) Overview of our method
Figure 1: The illustration of our proposed DCL model: (a) shows the overall architecture: the input videos with audiios (b) are first encoded by the corresponding visual and audio encoders, then the Disentangled Sequence Encoder (c) is used to decouple video features into static and dynamic factors by LSTM-based VAE. Through the Counterfactual Learning Module (d), we construct the affinity matrix  $A$  as a confounder and obtain the prediction  $\hat{Y}_{X,A_X}$  and counterfactual result  $\hat{Y}_{X,A_X^*}$ . Finally, we achieve  $\hat{Y}_{TIE}$  by subtracting these two results and optimizing the model by cross-entropy loss.
(d) Counterfactual Learning Module
then output inaccurate answers. In recent years, counterfactual thinking and causal reasoning have been widely applied in visual explanations [26, 27, 28, 29], scene graph generation [30, 31, 32, 33, 34], video understanding [35, 36, 37, 38, 39] and image recognition [31]. Recently, there has been some work on audio-visual video question answering [40, 41, 42], but they generally focus on cross-modal modeling. Our approach, in contrast, focuses on constructing physical knowledge relationships among different samples and using them as confounders in causal reasoning.
Figure 1(a) illustrates the overall framework of our proposed DCL. First, the model extracts features of input videos and the corresponding audio by visual and audio encoders, respectively. Then, a newly designed Disentangled Sequence Encoder (Sec. 3.2) is used to decouple the video into static and dynamic factors. Subsequently, the raw and intervened multi-modal features are obtained through the Counterfactual Learning Module (Sec. 3.3), and then fused with the question feature. Finally, the prediction result is obtained by optimizing the classification results based on the raw and intervened features (Sec. 3.4).
Problem Formulation. As shown in Figure 1(b), given a pair of videos (i.e.,  $< v_{1}, v_{2} >$ ) with the corresponding audios (i.e.,  $< a_{1}, a_{2} >$ ), where each video contains an object, and our task is to select a more appropriate object from the video inputs to answer the questions (i.e.,  $q$ ) about the physical commonsense. We define the estimated answer as  $\hat{Y}$  and the ground-truth answer as  $Y$  in the dataset. During the pre-processing, we denoted the extracted features of audio, video, and question text as  $X^{a}$ ,  $X^{v}$ , and  $X^{t}$ , respectively, where  $X^{a}, X^{v}, X^{t} \in \mathbb{R}^{d}$ ,  $d$  denotes the feature dimension. Especially,
we capture the audio feature and text feature as non-sequence while extracting the video feature as sequence data as  $X^{v} = \{X_{1}^{v},X_{2}^{v},\dots ,X_{T}^{v}\}$ , where  $T$  refers to the number of video frames.
3.2 Disentangled Sequential Encoder
The proposed Disentangled Sequential Encoder, as shown in Figure 1(c), can separate the static and dynamic factors for the multi-modal data (we take the video data as the example in the following), which extends the traditional sequential variational autoencoder by incorporating the mutual information term.
Specifically, we assume the latent representations of the input video's feature  $X_{1:T}^{v}$  can divide into the static factor  $s$  and the dynamic factors  $z_{1:T}$ , where  $z_{t}$  is the latent dynamic representation at time step  $t$ . Following [43, 44], we assume such two factors are independent of each other, i.e.,  $p(s,z_{1:T}) = p(s)p(z_{1:T})$ , where  $p(\cdot)$  represents the probability distribution. And  $z_{i}$  depends on  $z_{<i} = \{z_0,z_1,\dots,z_{i-1}\}$ , where  $z_0 = 0$  and the reconstruction of  $x_{i}^{3}$  is independent of other frames conditioned on  $z_{i}$  and  $s$ . Hence we need to learn a posterior distribution  $q(z_{1:T},s|x_{1:T})$  where the two factors are disentangled formulated as:
$$
q \left(z _ {1: T}, s \mid x _ {1: T}\right) = q \left(z _ {1: T} \mid x _ {1: T}\right) q (s \mid x _ {1: T}) = q (s \mid x _ {1: T}) \prod_ {i = 1} ^ {T} q \left(z _ {i} \mid z _ {<   i}, x _ {\leq i}\right). \tag {1}
$$
We model the posterior distribution by Bi-LSTM [45], where  $q(z_{i}|z_{<i},x_{\leq i})$  is conditioned on the entire time series by taking the hidden states as the input, and  $q(s|x_{1:T})$  can be computed by input  $x_{1:T}$ . Then we sample the two decoupled factors  $s$  and  $z_{1:T}$  by utilizing the distribution  $q(s|x_{1:T})$  and  $q(z_{i}|z_{<i},x_{\leq i})$  via the reparameterization trick [46]. Afterward, we use the obtained two disentangled factors to reconstruct the  $x_{1:T}$  based on a VAE-based decoder [44]. The prior of static factor  $s$  and dynamic factor  $z_{i}$  can be defined as the Gaussian distribution with  $\mathcal{N}(0,I)$  and  $\mathcal{N}(\mu (z_{<i}),\sigma^2 (z_{<i}))$ , respectively, where  $\mu (\cdot)$  and  $\sigma (\cdot)$  are modeled by Bi-LSTM. The reconstruction process can be formalized by the factorization:
$$
p \left(x _ {1: T}, s, z _ {1: T}\right) = p (s) \prod_ {i = 1} ^ {T} p \left(z _ {i} \mid z _ {<   i}\right) p \left(x _ {i} \mid z _ {i}, s\right). \tag {2}
$$
Furthermore, we introduce the mutual information to encourage the disentangled two factors mutually exclusive and incorporate non-parameter contrastive estimation into the standard loss function for learning latent representations [43, 47], which can be formulated as:
$$
\mathcal {C} \left(z _ {1: T}\right) = \mathbb {E} _ {p _ {D}} \log \frac {\phi \left(z _ {1 : T} , x _ {1 : T} ^ {+}\right)}{\phi \left(z _ {1 : T} , x _ {1 : T} ^ {+}\right) + \sum_ {j = 1} ^ {n} \phi \left(z _ {1 : T} , x _ {1 : T} ^ {j}\right)}, \tag {3}
$$
where  $x^{+}$  means the video has the same object as "positive example", while  $x^{j}, j = \{1,2,\dots,n\}$  is a set of  $n$  "negative" videos that have different objects. In order to alleviate the high dimensionality curse [48, 49], we use  $\phi(z_{1:T}, x_{1:T}^{+}) = \exp(sim(z_{1:T}, x_{1:T}^{+}) / \tau)$ , where  $sim(\cdot, \cdot)$  means cosine similarity function and  $\tau = 0.5$  is a temperature parameter.  $\mathcal{C}(s)$  can be calculated in the similar way.
Following [44], we use content augmentation by randomly shuffling the time steps of the video, and performing motion augmentation via the Gaussian blur [49]. The results can be denoted as  $\mathcal{C}(z_{1:T}^m)$  and  $\mathcal{C}(s^m)$ , where  $z_{1:T}^m$  and  $s^m$  are the augmented data of  $z_{1:T}$  and  $s$ , respectively. The MI term can be formulated as follows:
$$
M I \left(z _ {1: T}; x _ {1: T}\right) \approx \frac {1}{2} \left(\mathcal {C} \left(z _ {1: T}\right) + \mathcal {C} \left(z _ {1: T} ^ {m}\right)\right), \quad M I (s; x _ {1: T}) \approx \frac {1}{2} \left(\mathcal {C} (s) + \mathcal {C} \left(s ^ {m}\right)\right). \tag {4}
$$
The objective function can be formulated by adding MI terms to the vanilla ELBO as follows:
$$
\begin{array}{l} \mathcal {L} _ {D S E} = \mathbb {E} _ {q (z _ {1: T}, s | x _ {1: T})} [ - \sum_ {t = 1} ^ {T} \log p (x _ {t} | s, z _ {t}) ] + \gamma \cdot [ K L (q (s | x _ {1: T}) | | p (s)) \\ + \sum_ {t = 1} ^ {T} K L (q \left(z _ {t} \mid x _ {\leq t}\right) \mid \mid p \left(z _ {t} \mid z _ {<   t}\right)) ] - \alpha \cdot M I \left(z _ {1: T}; x _ {1: T}\right) - \beta \cdot M I (s; x _ {1: T}) + \theta \cdot M I \left(z _ {1: T}; s\right), \tag {5} \\ \end{array}
$$
For the sake of simplicity, in the following content, we will use  $x_{i}$  to represent  $X_{i}^{v}$ .
where  $\gamma, \alpha, \beta$  and  $\theta$  are hyper-parameters. The full proof can be found in [44].
3.3 Counterfactual Learning Module
In this module, we obtain the decoupled static and dynamic factors  $s$  and  $z_{1:T}$  after performing the Disentangled Sequential Encoder. Then, these factors with the corresponding audio features are sent to construct physical knowledge relationships. Meanwhile, we utilize the counterfactual relation intervention to improve knowledge learning.
Physical Knowledge Relationship. Inspired by Knowledge Graph [50, 51], the physical knowledge contained in different samples may have a certain correlation. Therefore, we propose to model such implicit relationships as a graph structure and draw the affinity matrix  $A$  to represent this physical knowledge relationship among various objects. Similarly, we build an affinity matrix for audio features or other modal data. With the well-constructed affinity matrix  $A$ , we can enhance the static, dynamic, and audio features denoted as  $X_{s}^{v}, X_{z}^{v}$ , and  $X^{a}$  through passing the message and transferring across various samples, formulated as:
$$
X = \left[ \begin{array}{c c c} X ^ {a} & X _ {s} ^ {v} & X _ {z} ^ {v} \end{array} \right], \qquad A _ {X} = \left[ \begin{array}{c c c} A _ {X ^ {a}} & A _ {X _ {s} ^ {v}} & A _ {X _ {z} ^ {v}} \end{array} \right], \qquad F = A _ {X} X ^ {\top}, \qquad (6)
$$
where  $A_{X}$  represents the augmented matrix composed of three affinity matrices, while  $F$  refers to the transferred features, and  $\top$  denotes the transpose of a matrix. To achieve  $A$ , we take  $A_{X_s^v}$  as an example: we first compute the similarity matrix based on the static features and filter out the noisy relationships by defining a near neighbor chosen function as  $\mathcal{T}(\cdot ,k)$  to retain the top-k values in each row, and then adopt the Laplacian matrix  $D^{-1}$  of  $\mathcal{S}'$  to normalize the total affinities as the following:
$$
\mathcal {S} ^ {i, j} = \exp (\frac {\operatorname {s i m} (x _ {i} , x _ {j})}{\tau}), x _ {i}, x _ {j} \in X _ {s} \quad \mathcal {S} ^ {\prime} = \mathcal {T} (\mathcal {S}, k), \quad A _ {X _ {s}} = D ^ {- 1} \cdot \mathcal {S} ^ {\prime}, \tag {7}
$$
where  $sim(\cdot, \cdot)$  is the cosine similarity. The calculation of  $A_z^v$  and  $A^a$  is similar to  $A_s^v$ .
Counterfactual Relation Intervention. To add additional supervision of the affinities  $A_{X}$ , we present to highlight the role of the object's physical knowledge relationship during the optimization. First, we express our method as a Structural Causal Model (SCM) [52, 53] as shown in Figure 1(d), and then introduce causal inference into our method.  $\hat{Y}$  represents the final classification output of the model, which is obtained by feeding forward F input into the fusion model and classifier:
$$
\hat {Y} _ {X, A _ {X}} = C L S \left(F u s i o n \left(F _ {1}, F _ {2}, X ^ {t}\right)\right), \tag {8}
$$
where  $F_{1}$  and  $F_{2}$  denote the fused feature of input video pair  $v_{1}$  and  $v_{2}$ , and  $X^{t}$  denotes the feature of the question text. 'CLS' and 'Fusion' represent the classifier and fusion model, respectively. Obviously, the process of deriving the output  $\hat{Y}$  from input  $X$  can be regarded as two types of effects: one is the direct effect  $X\rightarrow \hat{Y}$ , and the other is an indirect type  $X\rightarrow A_X\rightarrow \hat{Y}$ . Our final loss function is to maximum likelihood estimation, which will have an end-to-end effect on both types of effects, resulting in insufficient enhancement of  $A$  in the indirect effects path. Thus we leverage the Total Indirect Effect (TIE) [54] here to underline the  $A_{X}$ :
$$
\hat {Y} _ {T I E} = \hat {Y} _ {X, A _ {X}} - \mathbb {E} _ {X ^ {*}} \left[ \hat {Y} _ {X, A _ {X ^ {*}}} \right], \tag {9}
$$
where  $\hat{Y}_{X,A_{X^{*}}}$  refers to calculating the results by replacing the original affinity  $A_{X}$  to a intervened one  $A_{X^{*}}$ , and  $X^{*}$  is the given intervened inputs. Note that  $\hat{Y}_{X,A_{X^{*}}}$  cannot happen in the real world because features  $X$  and affinities  $A_{X^{*}}$  come from different  $X$  and  $X^{*}$ , which is called counterfactual intervention. Therefore, we modify  $\hat{Y}_{X,A_X}$  to  $\hat{Y}_{X,A_{X^{*}}}$  is equal to keeping all features fixed but only changing the affinity  $A$ , which can highlight the critical effect of  $A$ . We compute the expectation of that effect to get the more stable one, and the intervened input features  $X^{*}$  are sampled by a Gaussian distribution:
$$
X ^ {*} = X _ {\sigma} \cdot W + X _ {\mu}, \tag {10}
$$
where  $W$  is the standard random vector whose dimension is as same as  $X$ , and the mean  $X_{\mu}$  and standard deviation  $X_{\sigma}$  are both learned by the re-parameterization trick in the end-to-end learning.
3.4 Fusion Model and Optimization
As we have stated, our approach is a plug-and-play module that can be directly integrated into various multimodal fusion methods. We take Late fusion [55] as an example to illustrate the structure of fusion models, as well as the optimization function of our DCL. The fusion model can be expressed in the following formula:
$$
F u s i o n \left(F _ {1}, F _ {2}, X ^ {t}\right) = M L P (C o n c a t (M L P (C o n c a t \left(F _ {1}, F _ {2}\right)), X ^ {t})) \tag {11}
$$
where  $\text{Concat}$  represents row-wise concatenation, and  $MLP$  represents multi-layer perception.
Finally, our final optimization goal can be expressed as:
$$
\mathcal {L} _ {\text {t o t a l}} = \mathcal {L} _ {\text {D S E}} + \mathcal {L} _ {\text {T I E}}, \tag {12}
$$
where  $\mathcal{L}_{DSE}$  denote the optimization function we described in Eq. (5) and  $\mathcal{L}_{TIE}$  denotes the cross-entropy loss of  $\hat{Y}_{X,A_X}$  in Eq. (9) given the ground-truth  $Y$ .
Dataset. The Physical Audiovisual CommonSense Reasoning Dataset (PACS) [2] is a collection of 13,400 question-answer pairs designed for testing physical commonsense reasoning capabilities. The dataset contains 1,377 unique physical commonsense questions spanning various physical properties and is accompanied by 1,526 videos and audio clips downloaded from YouTube. The total number of data points in the PACS dataset is 13,400, and for PACS-Material it is 4,349. Following [2], We divide PACS into 11,044/1,192/1,164 as train/val/test set, each of which contains 1,224/150/152 objects respectively. We partitioned the PACS-Material subset into 3,460/444/445 for train/val/test under the same object distribution as PACS. To ensure a fair evaluation of models, we evaluate our method on both the entire dataset and a subset focused on material-related problems, reporting the results separately for each subset during testing.
Evaluation Metric. We follow the prior work [2] and use the accuracy to evaluate both PACS and PACS-material subsets. To evaluate the effectiveness of our static factor via disentanglement learning, we measure the accuracy of the PACS-material subset using only the static factor. In our experiments, we report results by running the experiments five times and taking the average following [2].
Implementation. We implement our proposed model with PyTorch on two NVIDIA RTX 3090 GPUs. Specifically, we downsampled each video to  $T = 8$  frames during pre-processing and set the feature dimension as  $d = 256$ . In the Disentangled Sequence Encoder, we used a hidden layer size of 256 for Bi-LSTM. During optimization, we set the batch size as 64, which consisted of 64 video pairs and the corresponding questions. In the Counterfactual Learning Module,  $\tau = 2$  and  $k = 5$  were used when calculating similarities and constructing the physical knowledge relationships. More details about hyper-parameters analysis can be found in our supplementary.
Compared Baseline Methods. To demonstrate the effectiveness of our method across multiple models, we apply it to the following baselines: 1) Late fusion [55], which adopts independent text, image, audio, and video encoders to obtain features, and the unimodal embeddings are concatenated and passed through a linear layer to create multimodal embeddings for prediction. 2) CLIP/AudioCLIP [56, 57], which embed video, text, and audio data into a shared vector space using CLIP and AudioCLIP, and a linear layer is used to create multimodal embeddings for prediction. Notes that CLIP is unable to extract audio features, so we did not use audio in our experiments with CLIP. 3) UNITER [58], which is a pre-trained image and text model for four different image-text tasks and performs well on tasks such as NLVR2 [59]. As for all of the above-mentioned baseline methods, we adopt the reported parameters following their corresponding paper. Note that another state-of-the-art Merlot Reserve [60] is a large-scale multimodal pretraining model that learns joint representations of audio, visual, and language by guiding the new pretraining objectives trained on TPU-v3-512. However, due to the restriction of experimental resources, we are unable to test the performance of DCL with this model and leave it as future work.