SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Yuzhang Shang $^{1}$ , Zhihang Yuan $^{2}$ , Yan Yan $^{1*}$
$^{1}$ Department of Computer Science, Illinois Institute of Technology
$^{2}$ Huomo AI
yshang4@hawk.iit.edu, zhihang.yuan@huomo.ai, yyan34@iit.edu
Dataset distillation (DD) aims to synthesize a small dataset whose test performance is comparable to a full dataset using the same model. State-of-the-art (SoTA) methods optimize synthetic datasets primarily by matching heuristic indicators extracted from two networks: one from real data and one from synthetic data (see Fig. 1, Left), such as gradients and training trajectories. DD is essentially a compression problem that emphasizes maximizing the preservation of information contained in the data. We argue that well-defined metrics which measure the amount of shared information between variables in information theory are necessary for success measurement but are never considered by previous works. Thus, we introduce mutual information (MI) as the metric to quantify the shared information between the synthetic and the real datasets, and devise MIM4DD numerically maximizing the MI via a newly designed estimizable objective within a contrastive learning framework to update the synthetic dataset. Specifically, we designate the samples in different datasets that share the same labels as positive pairs and vice versa negative pairs. Then we respectively pull and push those samples in positive and negative pairs into contrastive space via minimizing NCE loss. As a result, the targeted MI can be transformed into a lower bound represented by feature maps of samples, which is numerically feasible. Experiment results show that MIM4DD can be implemented as an add-on module to existing SoTA DD methods.
Deep learning has remarkably successful performance in computer vision tasks [25, 6], but most deep-learning-based methods require enormous amounts of data followed by extensive training resources. For example, in order to train a CLIP model [31] in a self-supervised manner [8], a high-resolution dataset with more than 10 million images are collected for training, which consumes tens of hundreds of GPU hours.
A straightforward solution to eliminate the reliance on training DNNs on data is to construct small training sets. Before the deep learning era, coreset or subset selection is the most prevalent paradigm, in which one can obtain a subset of salient data points to represent the original dataset of interest [1, 12, 3
Figure 1: (Left) General framework of previous SoTA DD methods: matching heuristic indicators extracted by networks from real and synthetic datasets; (Right) the motivation of MIM4DD: maximizing the mutual information between two datasets for optimizing the synthetic dataset.
[4]. In the era of deep learning, in contrast to
previously mentioned selection-based methods, Dataset Distillation (DD), also known as Dataset Condensation [41, 46, 40, 5], offers a revolutionary paradigm to synthesize a small dataset using gradient descent algorithms [33], as shown in Fig.1 (Left). In DD, the key question is how to define metrics to measure the distance between synthetic and real datasets. Only by optimizing a well designed distance metric can gradient descent be properly applied to update synthetic data. To answer this question, researchers design several distance algorithms. For example, Zhao et al. [46] measure distance between the batch gradients of the synthetic samples and original ones; Wang et al. [40] use the similarity between the feature maps extracted by networks on different datasets; Cazenavette et al. [5] resort to the MSE between the training trajectories of networks on real and synthetic datasets.
Dataset distillation task is essentially a compression problem with a strong emphasis on maximizing the preservation of information contained in the data [41, 19, 11]. Admittedly, previous heuristic-designed distance metrics have achieved promising performance. However, we argue that previous works neglect the well-defined distributional metric in information theory, which is necessary for success measurement. Specifically, if we define a dataset's samples as variables, it is imperative that the high-level distributional properties of these variables from synthetic and real datasets (e.g., correlations and dependencies between two datasets) should be captured and utilized to guide the update of synthetic datasets. Motivated by this notion, we introduce Mutual Information (MI), a well-formulated metric in information theory for dataset distillation. In detail, MI quantifies the information amount shared by the real and synthetic datasets. In contrast to the aforementioned works focusing on aligning the indicators extracted by different neural networks, MI can naturally capture non-linear statistical dependencies between variables and be used as a measure of true dependence, which is important information in data compression [4, 39, 36].
Based on MI metric, we propose a novel method, termed Mutual Information Maximization for Dataset Distillation (MIM4DD). In particular, we first formulate DD as a problem of MI maximization between two datasets. Then, we derive a numerically feasible lower bound and maximize this lower bound via contrastive learning [15, 18, 8]. Finally, we design a highly effective optimization strategy for the dataset distillation task using contrastive estimation for MI maximization. In this way, contrastive learning theoretically leads to the targeted MI maximization and also contributes to the inter-class decorrelation of synthetic samples. In other words, MIM4DD is built upon a contrastive learning framework to synthesize the small dataset, where samples from the synthetic dataset are pulled closer to the counterparts with the identical label in the real dataset, and pulled further away from the ones with different labels in the contrastive space. To the best of our knowledge, it is the first work aiming at MIM of the datasets over the DD task within a contrastive learning framework.
Overall, the contributions of this paper are three-fold: (i) To distill information from a large real dataset to a small synthetic dataset under a well-defined metric, we formulate the DD task as an MI maximization problem. To the best of our knowledge, this is the first work to introduce MI into the literature of DD. (ii) To maximize the targeted MI, we derive a numerically feasible lower bound and maximize it via contrastive learning. In this way, the heterogeneity in the generated images gets enhanced, which further improves the performance of the DD method. (iii) Experimental results show that our method outperforms existing SoTA methods. Importantly, our method can be implemented as a plug-and-play module for existing methods.
In this section, we demonstrate the methodology of Mutual Information Maximization for Dataset Distillation (MIM4DD). Firstly, we briefly revisit the general framework in previous SoTA DD methods, and we illustrate the MI background. Secondly, we model the DD problem within the literature on mutual information maximization (MIM). Then, we convert the targeted but numerical inaccessible MIM goal into a learnable object optimization problem, and solve the problem via a newly designed MIM4DD loss. Finally, we discuss the potential insights of MIM4DD. Note that we only elaborate on the key derivations in this section due to the space limitation; detailed discussions and technical theorems can be found in the supplemental material.
Dataset Distillation. In short, the goal of dataset distillation (also dubbed dataset condensation) is to synthesize a small training set,  $\mathcal{D}_{syn} = \{\mathbf{x}_{syn}^j,\mathbf{y}^j\}_{j = 1}^M$  such that models trained on this synthetic
dataset can have comparable performance as models (with the same architecture) trained on the large real set,  $\mathcal{D}_{real} = \{\mathbf{x}_ {real}^i,\mathbf{y}^i\}_{i = 1}^N$ , in which  $M\ll N$ . In this way, after obtaining the targeted synthetic data, the training process of DNNs can be largely accelerated. In this premise, we review some representative DD methods [43, 5, 40], which generally propose to enforce the behaviors of models trained on real and synthetic datasets to be similar. The core idea can be formalized in a form of an optimization problem:
$$
\mathcal {D} _ {s y n} ^ {\star} = \arg \min  _ {\mathcal {D} _ {s y n}} \mathbb {E} _ {\mathbf {x} _ {s y n} \sim \mathcal {D} _ {s y n}} \mathbb {E} _ {\mathbf {x} _ {r} \sim \mathcal {D} _ {r e a l}} \operatorname {D i s t} \left(f \left(\mathbf {x} _ {r e a l}; \theta^ {\star}\right), f \left(\mathbf {x} _ {s y n}; \gamma^ {\star}\right)\right), \tag {2.1}
$$
in which  $\theta^{\star}$  and  $\gamma^{\star}$  are the learned parameters of networks trained on the real dataset  $D_{real}$  and the synthetic dataset  $D_{syn}$ , respectively; and the distance function  $\mathrm{Dist}(\cdot, \cdot)$  is specifically developed to measure the similarity of two networks across datasets. With aligning the networks, one can optimize the synthetic data. Note that designing the distance functions aligning the networks trained on different datasets to optimize the synthetic dataset is the core of previous DD methods, e.g., CAFE [40] uses the MSE distance of feature maps, and MTT [5] calculates the difference of training trajectories as distance functions.
Although those heuristically designed distance functions lead to promising results, we argue that well-defined metrics in information theory for measuring the amount of shared information between variables have never been considered, as DD is essentially a compression problem with a different emphasis on the information contained in the data.
Mutual Information and Contrastive Learning. Mutual information quantifies the amount of information obtained about one random variable by observing the other random variable. It is a dimensionless quantity with (generally) units of bits, and can be considered as the reduction in uncertainty about one random variable given knowledge of another. High mutual information indicates a large reduction in uncertainty and vice versa [23]. Strictly, for two discrete variables  $\mathbf{X}$  and  $\mathbf{Y}$ , their mutual information (MI) can be defined as [23]:
$$
I (\mathbf {X}, \mathbf {Y}) = \sum_ {x, y} P _ {\mathbf {X Y}} (x, y) \log \frac {P _ {\mathbf {X Y}} (x , y)}{P _ {\mathbf {X}} (x) P _ {\mathbf {Y}} (y)}, \tag {2.2}
$$
where  $P_{\mathbf{X}\mathbf{Y}}(x,y)$  is the joint distribution,  $P_{\mathbf{X}}(x) = \sum_yP_{\mathbf{X}\mathbf{Y}}(x,y)$  and  $P_{\mathbf{Y}}(y) = \sum_xP_{\mathbf{X}\mathbf{Y}}(x,y)$  are the marginals of  $\mathbf{X}$  and  $\mathbf{Y}$ , respectively.
In the content of DD, we would like them to share as much information as possible. Theoretically, considering the samples in real and synthetic datasets as two random variables, the MI between those two variables should be maximized. Recently, contrastive learning has been proven an effective approach to maximize MI. Many methods based on contrastive loss for self-supervised learning are proposed, such as [18], [29], [42]. These methods are theoretically based on NCE [15] and InfoNCE [18]. Essentially, the key concept of contrastive learning is to pull representations in positive pairs close and push representations in negative pairs apart in a contrastive space. Thus the major obstacle for modeling problems in a contrastive way is to define the negative and positive pairs. In this work, we also resort to a contrastive learning framework for maximizing our targeted MI. Meanwhile, we illustrate how we formulate DD as a MI maximization problem, and how we solve this targeted problem within the contrastive learning framework.
2.2 MIM4DD: Mutual Information Maximization for Dataset Distillation
In this section, we first formalize the idea of maximizing the MI between the distributions of real and synthesized data, via constructing a contrastive task based on Noise-Contrastive Estimation (NCE). Specifically, we derive a novel MIM4DD loss to distill task-specific information from real data  $\mathcal{D}_{real}$  to synthetic data  $\mathcal{D}_{syn}$ , where NCE is introduced to avoid the direct MI computation by estimating it with its lower bound in Eq.2.14. From the perspective of NCE, straight-forwardly, the real and synthesized samples from the same class can be pulled close, and samples from different classes can be pushed away, which corresponds to the core idea of contrastive learning.
Problem Formulation: Ideally, for variable  $\mathbf{X}_{real}$  representing the samples in real data and  $\mathbf{X}_{syn}$  in the synthetic data, we desire to maximize the MI between  $\mathbf{X}_{real}$  and  $\mathbf{X}_{syn}$  in terms of  $\mathbf{X}_{syn}$ , i.e.,
$$
\text {T a r g e t e d} \mathbf {X} _ {s y n} ^ {\star} = \arg \max  _ {\mathbf {X} _ {s y n}} I \left(\mathbf {X} _ {\text {r e a l}}, \mathbf {X} _ {s y n}\right). \tag {2.3}
$$
In this way, synthetic data  $\mathcal{D}_{syn}^{\star}$  and fixed real data  $\mathcal{D}_{real}^{\star}$  can share maximal information. However, directly approaching this goal is unattainable since the distributions of datasets themselves are absurd to estimate [15, 18]. To encounter this obstacle, let us include the networks trained on real and synthetic datasets. We define those networks in the form of  $K$ -layer Multi-Layer Perceptrons (MLPs). For simplification, we discard the bias term of those MLPs. Then the network  $f(\mathbf{x})$  can be denoted as:
$$
f \left(\mathbf {W} ^ {1}, \dots , \mathbf {W} ^ {K}; \mathbf {x}\right) = \left(\mathbf {W} ^ {K} \cdot \sigma \cdot \mathbf {W} ^ {K - 1} \cdot \dots \cdot \sigma \cdot \mathbf {W} ^ {1}\right) (\mathbf {x}), \tag {2.4}
$$
where  $\mathbf{x}$  is the input sample and  $\mathbf{W}^k:\mathbb{R}^{d_{k - 1}}\longmapsto \mathbb{R}^{d_k}(k = 1,\dots,K)$  stands for the weight matrix connecting the  $(k - 1)$ -th and the  $k$ -th layer, with  $d_{k - 1}$  and  $d_{k}$  representing the sizes of the input and output of the  $k$ -th network layer, respectively. The  $\sigma (\cdot)$  function performs element-wise activation operation on the input feature maps. Based on those predefined notions, the sectional MLP  $f^{k}(\mathbf{x})$  with the front  $k$  layers of the  $f(\mathbf{x})$  can be represented as:
$$
f ^ {k} \left(\mathbf {W} ^ {1}, \dots , \mathbf {W} ^ {k}; \mathbf {x}\right) = \left(\mathbf {W} ^ {k} \cdot \sigma \dots \sigma \cdot \mathbf {W} ^ {1}\right) (\mathbf {x}). \tag {2.5}
$$
The  $k$ -th layer's feature maps are  $\mathbf{A}_{syn}^k$  and  $\mathbf{A}_{real}^k$ ,  $(k \in \{1, \dots, K\})$ , where  $\mathbf{A}_{syn}^k = (\mathbf{a}_{syn}^{k,1}, \dots, \mathbf{a}_{syn}^{k,M})$  and  $\mathbf{A}_{real}^k = (\mathbf{a}_{real}^{k,1}, \dots, \mathbf{a}_{real}^{k,N})$  can be considered as a series of variables. Specifically, the feature map can be obtained by:
$$
\mathbf {a} _ {s y n} ^ {k, j _ {c r}} = f ^ {k} \left(\mathbf {x} _ {s y n} ^ {j}\right), j \in \{1, \dots , M \}, \quad \mathbf {a} _ {r e a l} ^ {k, i _ {c s}} = f ^ {k} \left(\mathbf {x} _ {r e a l} ^ {i}\right), i \in \{1, \dots , N \}. \tag {2.6}
$$
Here, we show the relationship of MI in data  $\mathbf{X}$  level and in feature maps  $\mathbf{A}$  level, i.e., the relationship between  $I(\mathbf{X}_{real},\mathbf{X}_{syn})$  and  $I(\mathbf{A}_{real},\mathbf{A}_{syn})$ . We utilize the in-variance of MI to understand it. The property can be illustrated as follows:
Theorem 1 (In-variance of Mutual Information): Mutual information is invariant under the reparametrization of the marginal variables. If  $X' = F(X)$  and  $Y' = G(Y)$  are homeomorphisms (i.e.,  $F(\cdot)$  and  $G(\cdot)$  are smooth uniquely invertible maps), then
$$
I (X, Y) = I \left(X ^ {\prime}, Y ^ {\prime}\right). \tag {2.7}
$$
Since each layer's mapping  $\mathbf{W}^k:\mathbb{R}^{d_{k - 1}}\longmapsto \mathbb{R}^{d_k}(k = 1,\dots ,K)$  can be considered as the smooth uniquely invertible maps Theorem 1. Combining this theorem with the definition of MI in Eq.2.2, we observe that the MI in the targeted data level is equivalent to MI in the feature maps level, i.e.,
$$
I \left(\mathbf {X} _ {\text {r e a l}}, \mathbf {X} _ {\text {s y n}}\right) = I \left(\mathbf {A} _ {\text {r e a l}} ^ {k}, \mathbf {A} _ {\text {s y n}} ^ {k}\right), (k = 1, \dots , K). \tag {2.8}
$$
More details and the proof of this theorem are in Supplemental Materials and [21].
Facilitated by this property, we are able to access the calculation of MI between real and synthetic datasets via the feature maps of networks trained on those datasets. In other words, Theorem 1 helps us transfer the targeted MI maximization in Eq.2.3 to a reachable MI in Eq.2.8 in the dataset distillation literature, i.e.,
$$
\text {A c c e s s i b l e M I :} \quad \arg \max  _ {\mathbf {X} _ {s y n}} \sum_ {k = 1} ^ {K} I \left(\mathbf {A} _ {\text {r e a l}} ^ {k}, \mathbf {A} _ {\text {s y n}} ^ {k}\right), \tag {2.9}
$$
in which  $\mathbf{A}_{syn}^{k} = f^{k}(\mathbf{X}_{syn})$  and  $\mathbf{A}_{real}^{k} = f^{k}(\mathbf{X}_{real})$ .
Apart from the theoretical derivation, intuitively, the corresponding variables  $(\mathbf{a}_{syn}^{k}, \mathbf{a}_{real}^{k})$  should share more information, i.e., MI of the same layer's output feature maps  $I(\mathbf{a}_{syn}^{k}, \mathbf{a}_{real}^{k})$  ( $k \in \{1, \dots, K\}$ ) should be maximized to enforce them mutually dependent. This motivation has also been testified from the perspective of KD [17, 13] by CRD [39] and WCoRD [7]. In those methods, the penultimate layer's feature maps of teacher and student are aligned in a contrastive learning manner to enhance the heterogeneity of representations, which can also be explained via MI maximization. However, MIM4DD is different from those methods (details is in Related Work).
To optimize the accessible Mutual Information (MI) as defined in Eq.2.9, we incorporate a contrastive learning framework into our targeted Dataset Distillation (DD) task. The fundamental principle of contrastive learning involves comparing varying perspectives of the data, typically under different data augmentations, to compute similarity scores [29, 18, 2, 16, 8]. This framework is suitable for our case, since the activations from real and synthetic datasets can be seen as two different views.
Figure 2: Feeding images from two datasets into two corresponding neural networks, and we obtain the three pairs of representations for each layer. We embed the representations into a contrastive space, then learn from the pair correlation with the contrastive learning task in Eq. 2.16. In this way, the heterogeneity in the generated images can be enhanced. Theoretically, we can realize the formulated mutual information maximization (MIM) in Eq. 2.9, which is equivalent to the targeted MIM in Eq. 2.3.
(Definition: Positive and Negative Samples.) Let's denote each sample from the real and synthetic datasets as  $\{\mathbf{x}_{real}^{j_{cr}}\}$  ( $j_{cr} \in \{1, \dots, N\}$ ) and  $\{\mathbf{x}_{syn}^{i_{cs}}\}$  ( $i_{cs} \in \{1, \dots, M\}$ ), respectively. Here,  $c_{r}$  and  $c_{s} \in \{1, \dots, C\}$  represent the class labels of  $\mathbf{x}_{real}^{j_{cr}}$  and  $\mathbf{x}_{syn}^{i_{cs}}$ , respectively. We feed two batches of samples from two datasets to two different neural networks and obtain  $N \cdot M$  pairs of  $k$ -th layer's activations  $(\mathbf{a}_{syn}^{k,i_{cs}}, \mathbf{a}_{real}^{k,j_{cr}})$ , which can further be utilized for the contrastive learning task. We define a pair containing two activations corresponding to the same labels as the positive pair, i.e., if  $c_{r} = c_{s}$ ,  $(\mathbf{a}_{syn}^{k,i_{cs}}, \mathbf{a}_{real}^{k,j_{cr}})_{+}$  and vice versa. Consequently, there is  $\frac{M}{C} \cdot \frac{M}{C} \cdot C = \frac{1}{C} \cdot N \cdot M$  positive pairs, and thus  $(1 - \frac{1}{C}) \cdot N \cdot M$  negative pairs. The key concept of contrastive learning is to discriminate whether a given pair of activation  $(\mathbf{a}_{syn}^{k,i_{cs}}, \mathbf{a}_{real}^{k,j_{cr}})$  is positive or negative. In other words, it involves estimate the distribution  $P(D \mid \mathbf{a}_{syn}^{k,i_{cs}}, \mathbf{a}_{real}^{k,j_{cr}})$ , in which  $D$  is a scalar variable indicating whether  $c_{s} = c_{r}$  or  $c_{s} \neq c_{r}$ . Specifically,  $D = 1$  when  $c_{s} = c_{r}$ , and  $D = 0$  when  $c_{s} \neq c_{r}$ . However, we can not directly reach the distribution  $P(D \mid \mathbf{a}_{syn}^{k,i_{cs}}, \mathbf{a}_{real}^{k,j_{cr}})$  [15], and thus we introduce its corresponding variational form:
$$
q \left(D \mid \mathbf {a} _ {\text {r e a l}} ^ {k, j _ {c _ {r}}}, \mathbf {a} _ {\text {s y n}} ^ {k, i _ {c _ {s}}}\right). \tag {2.10}
$$
Intuitively,  $q(D \mid \mathbf{a}_{real}^{k,j_{c_r}}, \mathbf{a}_{syn}^{k,i_{c_s}})$  can be treated as a binary classifier, which can classify a given pair  $(\mathbf{a}_{syn}^{k,i_{c_s}}, \mathbf{a}_{real}^{k,j_{c_r}})$  into positive or negative. Importantly,  $q(D \mid \mathbf{a}_{real}^{k,i_{c_s}}, \mathbf{a}_{syn}^{k,j_{c_r}})$  can be estimated by some mature statistical methods, such as NCE [15] and InfoNCE [18].
Using the Bayes rule, the posterior probability of two activations from the positive pair can be formalized as:
$$
q \left(D = 1 \mid \mathbf {a} _ {\text {s y n}} ^ {k, i _ {c _ {s}}}, \mathbf {a} _ {\text {r e a l}} ^ {k, j _ {c _ {r}}}\right) = \frac {q \left(\mathbf {a} _ {\text {s y n}} ^ {k , i _ {c _ {s}}} , \mathbf {a} _ {\text {r e a l}} ^ {k , j _ {c _ {r}}} \mid D = 1\right) \frac {1}{C}}{q \left(\mathbf {a} _ {\text {s y n}} ^ {k , i _ {c _ {s}}}, \mathbf {a} _ {\text {r e a l}} ^ {k , j _ {c _ {r}}} \mid D = 1\right) \frac {1}{C} + q \left(\mathbf {a} _ {\text {s y n}} ^ {k , i _ {c _ {s}}}, \mathbf {a} _ {\text {r e a l}} ^ {k , j _ {c _ {r}}} \mid D = 0\right) \frac {C - 1}{C}}. \tag {2.11}
$$
The probability of activations from negative pair is  $q(D = 0 \mid \mathbf{a}_{syn}^{k,i_{cs}}, \mathbf{a}_{real}^{k,j_{cr}}) = 1 - q(D = 1 \mid \mathbf{a}_{syn}^{k,i_{cs}}, \mathbf{a}_{real}^{k,j_{cr}})$ . To simplify the NCE derivative, several works [15, 42, 39] build assumption about the dependence of the variables, we also use the assumption that the activations from positive pairs are dependent and the ones from negative pairs are independent, i.e.  $q(\mathbf{a}_{syn}^{k,i_{cs}}, \mathbf{a}_{real}^{k,j_{cr}} \mid D = 1) = P(\mathbf{a}_{syn}^{k,i_{cs}}, \mathbf{a}_{real}^{k,j_{cr}})$  and  $q(\mathbf{a}_{syn}^{k,i_{cs}}, \mathbf{a}_{real}^{k,j_{cr}} \mid D = 0) = P(\mathbf{a}_{syn}^{k,i_{cs}})P(\mathbf{a}_{real}^{k,j_{cr}})$ . Hence, the above equation can be simplified as:
$$
q (D = 1 \mid \mathbf {a} _ {\text {s y n}} ^ {k, i _ {c _ {s}}}, \mathbf {a} _ {\text {r e a l}} ^ {k, j _ {c _ {r}}}) = \frac {P \left(\mathbf {a} _ {\text {s y n}} ^ {k , i _ {c _ {s}}} , \mathbf {a} _ {\text {r e a l}} ^ {k , j _ {c _ {r}}}\right)}{P \left(\mathbf {a} _ {\text {s y n}} ^ {k , i _ {c _ {s}}} , \mathbf {a} _ {\text {r e a l}} ^ {k , j _ {c _ {r}}}\right) + P \left(\mathbf {a} _ {\text {s y n}} ^ {k , i _ {c _ {s}}}\right) P \left(\mathbf {a} _ {\text {r e a l}} ^ {k , j _ {c _ {r}}}\right) (C - 1)}. \tag {2.12}
$$
Performing logarithm to Eq.2.12 and arranging the terms, we can achieve
$$
\log q (D = 1 \mid \mathbf {a} _ {\text {s y n}} ^ {k, i _ {c s}}, \mathbf {a} _ {\text {r e a l}} ^ {k, j _ {c r}}) \leq \log \frac {P \left(\mathbf {a} _ {\text {s y n}} ^ {k , i _ {c s}} , \mathbf {a} _ {\text {r e a l}} ^ {k , j _ {c r}}\right)}{P \left(\mathbf {a} _ {\text {s y n}} ^ {k , i _ {c s}}\right) P \left(\mathbf {a} _ {\text {r e a l}} ^ {k , j _ {c r}}\right)} - \log (C - 1). \tag {2.13}
$$
Taking expectation of  $P(\mathbf{a}_{syn}^{k,i_{c_s}},\mathbf{a}_{real}^{k,j_{c_r}})$  on both sides, and combining Eq.2.2, we can transfer the MI into: