SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Paul Pu Liang<sup>1</sup>, Yun Cheng<sup>1,2</sup>, Xiang Fan<sup>1,3</sup>, Chun Kai Ling<sup>1,8</sup>, Suzanne Nie<sup>1</sup>, Richard J. Chen<sup>4,5</sup>, Zihao Deng<sup>6</sup>, Nicholas Allen<sup>7</sup>, Randy Auerbach<sup>8</sup>, Faisal Mahmood<sup>4,5</sup>, Ruslan Salakhutdinov<sup>1</sup>, Louis-Philippe Morency<sup>1</sup>
<sup>1</sup>CMU, <sup>2</sup>Princeton University, <sup>3</sup>UW, <sup>4</sup>Harvard Medical School, <sup>5</sup>Brigham and Women's Hospital, <sup>6</sup>University of Pennsylvania, <sup>7</sup>University of Oregon, <sup>8</sup>Columbia University pliang@cs.cmu.edu, yc6206@cs.princeton.edu
The recent explosion of interest in multimodal applications has resulted in a wide selection of datasets and methods for representing and integrating information from different modalities. Despite these empirical advances, there remain fundamental research questions: How can we quantify the interactions that are necessary to solve a multimodal task? Subsequently, what are the most suitable multimodal models to capture these interactions? To answer these questions, we propose an information-theoretic approach to quantify the degree of redundancy, uniqueness, and synergy relating input modalities with an output task. We term these three measures as the PID statistics of a multimodal distribution (or PID for short), and introduce two new estimators for these PID statistics that scale to high-dimensional distributions. To validate PID estimation, we conduct extensive experiments on both synthetic datasets where the PID is known and on large-scale multimodal benchmarks where PID estimations are compared with human annotations. Finally, we demonstrate their usefulness in (1) quantifying interactions within multimodal datasets, (2) quantifying interactions captured by multimodal models, (3) principled approaches for model selection, and (4) three real-world case studies engaging with domain experts in pathology, mood prediction, and robotic perception where our framework helps to recommend strong multimodal models for each application.
A core challenge in machine learning lies in capturing the interactions between multiple input modalities. Learning different types of multimodal interactions is often quoted as motivation for many successful multimodal modeling paradigms, such as contrastive learning to capture redundancy [54, 82], modality-specific representations to retain unique information [100], as well as tensors and multiplicative interactions to learn higher-order interactions [52, 58, 115]. However, several fundamental research questions remain: How can we quantify the interactions that are necessary to solve a multimodal task? Subsequently, what are the most suitable multimodal models to capture these interactions? This paper aims to formalize these questions by proposing an approach to quantify the nature (i.e., which type) and degree (i.e., the amount) of modality interactions, a fundamental principle underpinning our understanding of multimodal datasets and models [62].
By bringing together two previously disjoint research fields of Partial Information Decomposition (PID) in information theory [13, 41, 108] and multimodal machine learning [9, 62], we provide precise definitions categorizing interactions into redundancy, uniqueness, and synergy. Redundancy quantifies information shared between modalities, uniqueness quantifies the information present in only one of the modalities, and synergy quantifies the emergence of new information not previously present in either modality. A key aspect of these four measures is that they not only quantify interactions between modalities, but also how they relate to a downstream task. Figure 1 shows a depiction of these four measures, which we refer to as PID statistics. Leveraging insights from
neural representation learning, we propose two new estimators for PID statistics that can scale to high-dimensional multimodal datasets and models. The first estimator is exact, based on convex optimization, and is able to scale to features with discrete support, while the second estimator is an approximation based on sampling, which enables us to handle features with large discrete or even continuous supports. We validate our estimation of PID in 2 ways: (1) on synthetic datasets where PID statistics are known due to exact computation and from the nature of data generation, and (2) on real-world data where PID is compared with human annotation. Finally, we demonstrate that these estimated PID statistics can help in multimodal applications involving:
1. Dataset quantification: We apply PID to quantify large-scale multimodal datasets, showing that these estimates match common intuition for interpretable modalities (e.g., language, vision, and audio) and yield new insights in other domains (e.g, healthcare, HCI, and robotics).
2. Model quantification: Across a suite of models, we apply PID to interpret model predictions and find consistent patterns of interactions that different models capture.
3. Model selection: Given our findings from dataset and model quantification, a new research question naturally arises: given a new multimodal task, can we quantify its PID values to infer (a priori) what type of models are most suitable? Our experiments show success in model selection for both existing benchmarks and completely new case studies engaging with domain experts in computational pathology, mood prediction, and robotics to select the best multimodal model.
Finally, we make public a suite of trained models across 10 model families and 30 datasets to accelerate future analysis of multimodal interactions at https://github.com/qliang279/PID.
Let  $\mathcal{X}_i$  and  $\mathcal{Y}$  be sample spaces for features and labels. Define  $\Delta$  to be the set of joint distributions over  $(\mathcal{X}_1, \mathcal{X}_2, \mathcal{Y})$ . We are concerned with features  $X_1, X_2$  (with support  $\mathcal{X}_i$ ) and labels  $Y$  (with support  $\mathcal{Y}$ ) drawn from some distribution  $p \in \Delta$ . We denote the probability mass (or density) function by  $p(x_1, x_2, y)$ , where omitted parameters imply marginalization. Key to our work is defining estimators that given  $p$  or samples  $\{(x_1, x_2, y) : \mathcal{X}_1 \times \mathcal{X}_2 \times \mathcal{Y}\}$  thereof (i.e., dataset or model predictions), returns estimates for the amount of redundant, unique, and synergistic interactions.
2.1 Partial Information Decomposition
Information theory formalizes the amount of information that one variable provides about another [86]. However, its extension to 3 variables is an open question [34, 70, 93, 106]. In particular, the natural three-way mutual information  $I(X_{1};X_{2};Y) = I(X_{1};X_{2}) - I(X_{1};X_{2}|Y)$  [70, 93] can be both positive and negative, which makes it difficult to interpret. In response, Partial information decomposition (PID) [108] generalizes information theory to multiple variables by decomposing  $I_{p}(X_{1},X_{2};Y)$ , the total information 2 vari
Figure 1: PID decomposes  $I(X_{1},X_{2};Y)$  into redundancy  $R$  between  $X_{1}$  and  $X_{2}$ , uniqueness  $U_{1}$  in  $X_{1}$  and  $U_{2}$  in  $X_{2}$ , and synergy  $S$  in both  $X_{1}$  and  $X_{2}$ .
ables  $X_{1}, X_{2}$  provide about a task  $Y$  into 4 quantities (see Figure 1): redundancy  $R$  between  $X_{1}$  and  $X_{2}$ , uniqueness  $U_{1}$  in  $X_{1}$  and  $U_{2}$  in  $X_{2}$ , and synergy  $S$  that only emerges when both  $X_{1}$  and  $X_{2}$  are present. We adopt the PID definition proposed by Bertschinger et al. [13]:
$$
R = \max  _ {q \in \Delta_ {p}} I _ {q} \left(X _ {1}; X _ {2}; Y\right), \tag {1}
$$
$$
U _ {1} = \min  _ {q \in \Delta_ {p}} I _ {q} \left(X _ {1}; Y \mid X _ {2}\right), \quad U _ {2} = \min  _ {q \in \Delta_ {p}} I _ {q} \left(X _ {2}; Y \mid X _ {1}\right), \tag {2}
$$
$$
S = I _ {p} \left(X _ {1}, X _ {2}; Y\right) - \min  _ {q \in \Delta_ {p}} I _ {q} \left(X _ {1}, X _ {2}; Y\right), \tag {3}
$$
where  $\Delta_p = \{q \in \Delta : q(x_i, y) = p(x_i, y) \forall y \in \mathcal{Y}, x_i \in \mathcal{X}_i, i \in [2]\}$  and the notation  $I_p(\cdot)$  and  $I_q(\cdot)$  disambiguates mutual information under  $p$  and  $q$  respectively. The key lies in optimizing  $q \in \Delta_p$  to satisfy the marginals  $q(x_i, y) = p(x_i, y)$ , but relaxing the coupling between  $x_1$  and  $x_2$ :  $q(x_1, x_2)$  need not be equal to  $p(x_1, x_2)$ . The intuition behind this is that one should be able to infer redundancy and uniqueness given only access to  $p(x_1, y)$  and  $p(x_2, y)$ , and therefore they should only depend on  $q \in \Delta_p$ . Synergy is the only term that should depend on the coupling  $p(x_1, x_2)$ , and this is reflected
in (3) depending on the full  $p$  distribution. This definition enjoys several useful properties in line with intuition, as we will see in comparison with related frameworks for interactions below [13].
2.2 Related Frameworks for Feature Interactions
Information-theoretic definitions: Perhaps the first measure of redundancy in machine learning is co-training [14, 8, 23], where 2 variables are redundant if they are conditionally independent given the task:  $I(X_{1};X_{2}|Y) = 0$ . As a result, redundancy can be measured by  $I(X_{1};X_{2};Y)$ . The same definition of redundancy is used in multi-view learning [98, 94, 89] which further define  $I(X_{1};Y|X_{2})$  and  $I(X_{2};Y|X_{1})$  as unique information in  $X_{1}, X_{2}$ . However,  $I(X_{1};X_{2};Y)$  can be both positive and negative [51]. PID resolves this by separating  $R$  and  $S$  such that  $R - S = I(X_{1};X_{2};Y)$ , identifying that prior measures confound redundancy and synergy. This crucially provides an explanation for the distinction between mediation, where one feature conveys the information already in another (i.e.,  $R > S$ ), versus moderation, where one feature affects the relationship of other features (i.e.,  $S > R$ ) [10, 36]. Furthermore, if  $I(X_{1};X_{2};Y) = 0$  then existing frameworks are unable to distinguish between positive  $R$  and  $S$  canceling each other out.
Statistical measures: Other approaches have studied interaction measures via statistical measures, such as redundancy via distance between prediction logits using either feature [69], statistical distribution tests on input features [114, 7], or via human annotations [85]. However, it is unclear how to extend these definitions to uniqueness and synergy while remaining on the same standardized scale like PID provides. Also of interest are notions of redundant and synergistic interactions in human and animal communication [78, 79, 30, 85], which we aim to formalize.
Model-based methods: Prior research has formalized definitions of non-additive interactions [32] to quantify their presence [88, 101, 102, 46] in trained models, or used Shapley values on trained features to measure interactions [49]. Parallel research has also focused on qualitative visualizations of real-world multimodal datasets and models, such as DIME [68], M2Lens [105], and MultiViz [61].
3 Scalable Estimators for PID
PID as a framework for multimodality: Our core insight is that PID provides a formal framework to understand both the nature and degree of interactions involved when two features  $X_{1}$  and  $X_{2}$  are used for task  $Y$ . The nature of interactions is afforded by a precise decomposition into redundant, unique, and synergistic interactions, and the degree of interactions is afforded by a standardized unit of measure (bits). However, computing PID is a considerable challenge, since it involves optimization over  $\Delta_{p}$  and estimating information-theoretic measures. Up to now, analytic approximations of these quantities were only possible for discrete and small support [13, 41, 109] or continuous but low-dimensional variables [76, 81, 110]. Leveraging ideas in representation learning, Sections 3.1 and 3.2 are our first technical contributions enabling scalable estimation of PID for high-dimensional distributions. The first, CVX, is exact, based on convex optimization, and is able to scale to problems where  $|\mathcal{X}_i|$  and  $|\mathcal{Y}|$  are around 100. The second, BATCH, is an approximation based on sampling, which enables us to handle large or even continuous supports for  $X_{i}$  and  $Y$ . Applying these estimators in Section 4, we show that PID provides a path towards understanding the nature of interactions in datasets and those learned by different models, and principled approaches for model selection.
3.1 CVX: Dataset-level Optimization
Our first estimator, CVX, directly compute PID from its definitions using convex programming. Crucially, Bertschinger et al. [13] show that the solution to the max-entropy optimization problem:  $q^{*} = \arg \max_{q\in \Delta_{p}}H_{q}(Y|X_{1},X_{2})$  equivalently solves (1)-(3). When  $\mathcal{X}_i$  and  $\mathcal{V}$  are small and discrete, we can represent all valid distributions  $q(x_{1},x_{2},y)$  as a set of tensors  $Q$  of shape  $|\mathcal{X}_1|\times |\mathcal{X}_2|\times |\mathcal{V}|$  with each entry representing  $Q[i,j,k] = p(X_1 = i,X_2 = j,Y = k)$ . The problem then boils down to optimizing over valid tensors  $Q\in \Delta_p$  that match the marginals  $p(x_i,y)$ .
Given a tensor  $Q$  representing  $q$ , our objective is the concave function  $H_{q}(Y|X_{1},X_{2})$ . While Bertschinger et al. [13] report that direct optimization is numerically difficult as routines such as Mathematica's FINDMINIMUM do not exploit convexity, we overcome this by rewriting conditional entropy as a KL-divergence [38],  $H_{q}(Y|X_{1},X_{2}) = \log |\mathcal{V}| - KL(q||\tilde{q})$ , where  $\tilde{q}$  is an auxiliary product density of  $q(x_{1},x_{2})\cdot \frac{1}{|\mathcal{V}|}$  enforced using linear constraints:  $\tilde{q} (x_1,x_2,y) = q(x_1,x_2) / |\mathcal{V}|$ . Finally, optimizing over  $Q\in \Delta_p$  that match the marginals can also be enforced through linear constraints: the 3D-tensor  $Q$  summed over the second dimension gives  $q(x_{1},y)$  and summed over
Figure 2: We propose BATCH, a scalable estimator for PID over high-dimensional continuous distributions. BATCH parameterizes  $\tilde{q}$  using a matrix  $A$  learned by neural networks such that mutual information objectives over  $\tilde{q}$  can be optimized via gradient-based approaches over minibatches. Marginal constraints  $\tilde{q} \in \Delta_p$  are enforced through a variant of the Sinkhorn-Knopp algorithm on  $A$ .
the first dimension gives  $q(x_2, y)$ , yielding the final optimization problem:
$$
\underset {Q, \tilde {Q}} {\arg \max } K L (Q \| \tilde {Q}), \quad \text {s . t .} \quad \tilde {Q} \left(x _ {1}, x _ {2}, y\right) = Q \left(x _ {1}, x _ {2}\right) / | \mathcal {Y} |, \tag {4}
$$
$$
\sum_ {x _ {2}} Q = p \left(x _ {1}, y\right), \sum_ {x _ {1}} Q = p \left(x _ {2}, y\right), Q \geq 0, \sum_ {x _ {1}, x _ {2}, y} Q = 1. \tag {5}
$$
The KL-divergence objective is recognized as convex, allowing the use of conic solvers such as SCS [75], ECOS [27], and MOSEK [6]. Plugging  $q^{*}$  into (1)-(3) yields the desired PID.
Pre-processing via feature binning: In practice,  $X_{1}$  and  $X_{2}$  often take continuous rather than discrete values. We work around this by histogramming each  $X_{i}$ , thereby estimating the continuous joint density by discrete distributions with finite support. We provide more details in Appendix B.1.
3.2 BATCH: Batch-level Amortization
We now present BATH, our next estimator that is suitable for large datasets where  $\mathcal{X}_i$  is high-dimensional and continuous (  $|\mathcal{Y}|$  remains finite). To estimate PID given a sampled dataset  $\mathcal{D} = \{(x_1^{(j)},x_2^{(j)},y^{(j)})\}$  of size  $n$ , we propose an end-to-end model parameterizing marginal-matching joint distributions in  $\Delta_p$  and a training objective whose solution returns approximate PID values.
Simplified algorithm sketch: Our goal, loosely speaking, is to optimize  $\tilde{q} \in \Delta_p$  for objective (1) through an approximation using neural networks instead of exact optimization. We show an overview in Figure 2. To explain our approach, we first describe (1) how we parameterize  $\tilde{q}$  using neural networks such that it can be learned via gradient-based approaches, (2) how we ensure the marginal constraints  $\tilde{q} \in \Delta_p$  through a variant of the Sinkhorn-Knopp algorithm, and finally (3) how to scale this up over small subsampled batches from large multimodal datasets.
Parameterization using neural networks: The space of joint distributions  $\Delta$  is often too large to explicitly specify. To tackle this, we implicitly parameterize each distribution  $\tilde{q} \in \Delta$  using a neural network  $f_{\phi}$  that takes in batches of modalities  $\mathbf{X}_1 \in \widetilde{\mathcal{X}}_1^n$ ,  $\mathbf{X}_2 \in \widetilde{\mathcal{X}}_2^n$  and the label  $\mathbf{Y} \in \mathcal{Y}^n$  before returning a matrix  $A \in \mathbb{R}^{n \times n \times |\mathcal{Y}|}$  representing an (unnormized) joint distribution  $\tilde{q}$ , i.e., we want  $A[i][j][y] = \tilde{q}(\mathbf{X}_1[i], \mathbf{X}_2[j], y)$  for each  $y \in \mathcal{Y}$ . In practice,  $f_{\phi}$  is implemented via a pair of encoders  $f_{\phi(1)}$  and  $f_{\phi(2)}$  that learn modality representations, before an outer product to learn joint relationships  $A_y = \exp(f_{\phi(1)}(\mathbf{X}_1, y)f_{\phi(2)}(\mathbf{X}_2, y)^\top)$  for each  $y$ , yielding the desired  $n \times n \times |\mathcal{Y}|$  joint distribution. As a result, optimizing over  $\tilde{q}$  can be performed via optimizing over parameters  $\phi$ .
Respecting the marginal constraints: How do we make sure the  $\tilde{q}$ 's learned by the network satisfies the marginal constraints (i.e.,  $\tilde{q} \in \Delta_p$ )? We use an unrolled version of Sinkhorn's algorithm [24] which projects  $A$  onto  $\Delta_p$  by iteratively normalizing  $A$ 's rows and columns to sum to 1 and rescaling to satisfy the marginals  $p(x_i, y)$ . However,  $p(x_i, y)$  is not easy to estimate for high-dimensional continuous  $x_i$ 's. In response, we first expand  $p(x_i, y)$  into  $p(y|x_i)$  and  $p(x_i)$  using Bayes' rule. Since  $A$  was constructed by samples  $x_i$  from the dataset, the rows and columns of  $A$  are already distributed according to  $p(x_1)$  and  $p(x_2)$  respectively. This means that it suffices to approximate  $p(y|x_i)$  with unimodal classifiers  $\hat{p}(y|x_i)$  parameterized by neural networks and trained separately, before using Sinkhorn's algorithm to normalize each row to  $\hat{p}(y|x_1)$  and each column to  $\hat{p}(y|x_2)$ .
Objective: We choose the objective  $I_q(X_1; X_2; Y)$ , which equivalently solves the optimization problems in the other PID terms [13]. Given matrix  $A$  representing  $\tilde{q}(x_1, x_2, y)$ , the objective can be computed in closed form through appropriate summation across dimensions in  $A$  to obtain  $\tilde{q}(x_i)$ ,  $\tilde{q}(x_1, x_2)$ ,  $\tilde{q}(x_i|y)$ , and  $\tilde{q}(x_1, x_2|y)$  and plugging into  $I_{\tilde{q}}(X_1; X_2; Y) = I_{\tilde{q}}(X_1; X_2) - I_{\tilde{q}}(X_1; X_2|Y)$ . We maximize  $I_{\tilde{q}}(X_1; X_2; Y)$  by updating parameters  $\phi$  via gradient-based methods. Overall, each
Figure 3: Left to right: (a) Contour plots of the GMM's density for  $||\mu ||_2 = 2.0$ . Red line denotes the optimal linear classifier. (b) PID (Cartesian) computed for varying  $\angle \mu$  with respect to the  $x$  axis. (c) PID (Polar) for varying  $\angle \mu$ , with  $U_{1}$  and  $U_{2}$  corresponding to unique information from  $(r,\theta)$ . Plots (d)-(f) are similar to (a)-(c), but repeated for  $||\mu ||_2 = 1.0$ . Legend:  $\star(R)$ ,  $\triangle(U_1)$ ,  $\nabla(U_2)$ ,  $\bullet(S)$ ,  $\ddagger(\mathrm{Sum})$ . Observe how PID changes with the change of variable from Cartesian (b and e) to Polar (c and f), as well as how a change in  $||\mu ||_2$  can lead to a disproportionate change across PID (b vs e).
gradient step involves computing  $\tilde{q} = \mathrm{SINKHORN}_{\tilde{p}}(A)$ , and updating  $\phi$  to maximize (1) under  $\tilde{q}$ . Since Sinkhorn's algorithm is differentiable, gradients can be backpropagated end-to-end.
Approximation with small subsampled batches: Finally, to scale this up to large multimodal datasets where the full  $\tilde{q}$  may be too large to store, we approximate  $\tilde{q}$  with small subsampled batches: for each gradient iteration  $t$ , the network  $f_{\phi}$  now takes in a batch of  $m \ll n$  datapoints sampled from  $\mathcal{D}$  and returns  $A \in \mathbb{R}^{m \times m \times |\mathcal{V}|}$  for the subsampled points. We perform Sinkhorn's algorithm on  $A$  and a gradient step on  $\phi$  as above, as if  $\mathcal{D}_t$  was the full dataset (i.e., mini-batch gradient descent). While it is challenging to obtain full-batch gradients since computing the full  $A$  is intractable, we found our approach to work in practice for large  $m$ . Our approach can also be informally viewed as performing amortized optimization [3] by using  $\phi$  to implicitly share information about the full batch using subsampled batches. Upon convergence of  $\phi$ , we extract PID by plugging  $\tilde{q}$  into (1)-(3).
Implementation details such as the network architecture of  $f$ , approximation of objective (1) via sampling from  $\tilde{q}$ , and estimation of  $I_{\tilde{q}}(\{X_1, X_2\}; Y)$  from learned  $\tilde{q}$  are in Appendix B.3.
4 Evaluation and Applications of PID in Multimodal Learning
We design experiments to (1) understand PID on synthetic data, (2) quantify real-world multimodal benchmarks, (3) understand the interactions captured by multimodal models, (4) perform model selection across different model families, and (5) applications on novel real-world tasks.
4.1 Validating PID Estimates on Synthetic Data
Our first goal is to evaluate the accuracy of our proposed estimators with respect to the ground truth (if it can be computed) or human judgment (for cases where the ground truth cannot be readily obtained). We start with a suite of datasets spanning both synthetic and real-world distributions.
Synthetic bitwise features: We sample from a binary bitwise distribution:  $x_{1}, x_{2} \sim \{0, 1\}$ ,  $y = x_{1} \wedge x_{2}$ ,  $y = x_{1} \vee x_{2}$ ,  $y = x_{1} \oplus x_{2}$ . Each bitwise operator's PID can be solved exactly when the  $x_{i}$ 's and labels are discrete and low-dimensional [13]. Compared to the ground truth in Bertschinger et al. [13], both our estimators exactly recover the correct PID values (Table 1).
Table 1: Results on estimating PID on synthetic bitwise datasets. Both our estimators exactly recover the correct PID values as reported in Bertschinger et al. [13].
Task	OR	AND	XOR
PID	R	U1	U2	S	R	U1	U2	S	R	U1	U2	S
Exact	0.31	0	0	0.5	0.31	0	0	0.5	0	0	0	1
CVX	0.31	0	0	0.5	0.31	0	0	0.5	0	0	0	1
BATCH	0.31	0	0	0.5	0.31	0	0	0.5	0	0	0	1
Gaussian Mixture Models (GMM): Consider a GMM, where  $X_{1}, X_{2} \in \mathbb{R}$  and the label  $Y \in \{-1, +1\}$ , comprising two equally weighted standard multivariate Gaussians centered at  $\pm \mu$ , where  $\mu \in \mathbb{R}^2$ , i.e.,  $Y \sim \text{Bernoulli}(1/2)$ ,  $(X_{1}, X_{2})|Y = y \sim \mathcal{N}(y \cdot \mu, I)$ . PID was estimated by sampling 1e6 points, histogramming them into 50 bins spanning  $[-5, +5]$  to give  $p$ , and then applying the CVX estimator. We term this PID-Cartesian. We also compute PID-Polar, which are PID computed using polar coordinates,  $(r, \theta)$ . We use a variant where the angle  $\theta$  is given by the arctangent with principal values  $[0, \pi]$  and the length  $r \in \mathbb{R}$  could be negative.  $\theta$  specifies a line (through the origin), and  $r$  tells us where along the line the datapoint lies on.
Results: We consider  $||\mu ||_2 \in \{1.0, 2.0\}$ , where for each  $||\mu ||_2$ , we vary the angle  $\angle \mu$  that  $\mu$  makes with the horizontal axis. Our computed PID is presented in Figure 3. Overall, we find that the PID matches what we expect from intuition. For Cartesian, unique information dominates when