MetaDD: Boosting Dataset Distillation with Neural Network Architecture-Invariant Generalization

## Abstract

Abstract Dataset distillation (DD) entails creating a refined, compact distilled dataset from a large-scale dataset to facilitate efficient training. A significant challenge in DD is the dependency between the distilled dataset and the neural network (NN) architecture used. Training a different NN architecture with a distilled dataset distilled using a specific architecture often results in diminished trainning performance for other architectures. This paper introduces MetaDD, designed to enhance the generalizability of DD across various NN architectures. Specifically, MetaDD partitions distilled data into meta features (i.e., the data’s common characteristics that remain consistent across different NN architectures) and heterogeneous features (i.e., the data’s unique feature to each NN architecture). Then, MetaDD employs an architecture-invariant loss function for multi-architecture feature alignment, which increases meta features and reduces heterogeneous features in distilled data. As a low-memory consumption component, MetaDD can be seamlessly integrated into any DD methodology. Experimental results demonstrate that MetaDD significantly improves performance across various DD methods. On the Distilled Tiny-Imagenet with Sre2L (50 IPC), MetaDD achieves cross-architecture NN accuracy of up to 30.1%, surpassing the second-best method (GLaD) by 1.7%.

## 1 Introduction

Figure: (a) Tiny ImageNet
Refer to caption: x2.png

Neural networks (NNs) are driven by data and the capability of NNs depends on the scale and quality of the data [, ]. However, as the scale of data grows, the training cost of NNs also becomes increasingly high. As a compression technique for the dataset, dataset distillation (DD) [] is proposed to reduce the dataset size, which generates a small distilled dataset as a substitute. DD is particularly useful when memory and computational power are limited, such as on edge devices or in real-time applications. By leveraging DD, researchers can achieve high-performance model training with significantly reduced data and resource requirements.

One major challenge for DD is its lack of transferability across different NN architectures. When DD is used with specific NNs to generate distilled datasets, there is a significant performance drop when other NN architectures are trained on these distilled datasets. Existing solutions to this cross-architecture gap have various shortcomings. Some fail to integrate with vision transformers [], others only improve the performance of the NN architectures involved in the DD process [], and some impose high memory costs due to image generators []. These issues highlight the need for more robust and versatile DD techniques that maintain high cross-architectures performance.

Figure: Figure 3: The framework of MetaDD. Our method is designed to supervise the synthesis of data during training to ensure it exhibits low-variance CAMs across multiple pre-trained NNs.
Refer to caption: x4.png

To investigate the causes of DD’s cross-architecture gap, we visualize NN architecture biases in feature preferences using Class Activation Maps (CAMs) [, ]. CAMs highlight the regions of an input image that are most relevant to an NN’s prediction, showing which features are important for its decision-making process. We use the overlap of different architectures’ CAM regions as the decision-making consensus (meta feature), while every architecture’s non-overlap region represents its unique feature preference bias (heterogeneous feature). We focus on the reason for two differences: (1) The better cross-architecture performance of the original dataset than the distilled dataset. (2)The better distilled dataset performance of the distilled NN architecture than crossed NN architectures.

By exploring both differences, we find that distilled datasets have massive heterogeneous features of the distilled architecture and rare meta features. As shown in Figure 2: (1) Original images show substantial meta features across different architectures trained on the original dataset, whereas distilled images show almost no meta features. This meta feature comparison explains why original dataset performs more consistently across different NNs than distilled data. (2) Distilled images have significantly more heterogeneous features with the same architecture used for DD compared to crossed architectures. These heterogeneous features represent the same architecture can extract more semantic information from distilled data, which crossed NN architecture cannot capture, leading to cross-architecture gap.

Based on the above observations, we propose MetaDD to improve the cross-architecture generalization of DD. MetaDD begins with using an architecture-invariant loss to obtain and maximize the exposure of diverse features across different NN architectures. Then, MetaDD decouples heterogeneous and meta features by transferring distilled data’s CAMs to a common space.
By driving the evolution of distilled data towards maximizing meta features, MetaDD encourages to form a generalized consensus cross different NN architectures. MetaDD maintains low memory consumption by persistently freezing various NN architectures during DD. We conducted comprehensive experiments using DC, DM, MTT, and Sre2L as baselines. Incorporating MetaDD improved DD performance in cross-architecture training. MetaDD covers typical NN architectures, ensuring that even unconsidered models benefit from the meta features generated. On TinyImagenet and ILSVRC-2012, MetaDD has an average accuracy increase of 1.6% and 1.0% compared with GlaD [].

## 2 Related Work

DD [, , ] generates distilled datasets by aligning distilled data with the original data using a specific NN. This alignment is achieved through various techniques. For instance, model gradients [] are used to adjust the distilled data to match the gradient patterns of the original data. Similarly, the features [] extracted by the NN are aligned to ensure that the distilled data captures the same feature distributions as the original data. Moreover, parameter trajectories [] are tracked and matched, providing a dynamic way to align the evolving parameters during training. Additionally, kernel ridge regression statistics of the NNs [, ] are aligned to refine the distilled data further, ensuring that it statistically mirrors the original data from the kernel-based perspective.

Furthermore, the creation of novel data augmentation components [, , ] significantly enhances the distilled datasets. These components introduce new data transformations that enrich the variability of the distilled data. Reusable distillation paradigms [, , ] are also employed, which involve extracting and transferring knowledge from multiple models or datasets into the distilled dataset, thus improving its performance. Additionally, patches for fixing defects [, , , ] are developed and applied, addressing any inconsistencies or errors in the distilled data. These patches help maintain the distilled datasets’ integrity and accuracy.

### 2.1 Class Activation Mapping

CAM [, , , , , , , ] algorithms constitute an important class of methods in the field of deep learning, aimed at enhancing the interpretability of NNs.
The original CAM [] involves modifying the network architecture to connect directly from the global average pooling layer to the output layer, allowing the model to highlight important areas of the image for predictions of specific classes. Subsequent developments, such as Grad-CAM [], offer a universal solution that does not require modifications to the network architecture.
Additionally, variants such as Grad-CAM++ [] and Score-CAM [] have further improved the accuracy and robustness of the heatmaps.

Although CAM was initially designed for CNNs, researchers have begun exploring how similar concepts can be applied to transformer-based visual models. To achieve this application, CAMs are generated by analyzing the attention weights from the last layer []. Consequently, researchers have begun developing new techniques and approaches to improve the vision transformer CAM[, ], for example, by adjusting or combining attention weights from different layers, or developing specialized interpretative modules to produce clearer and more meaningful visual explanations. facilitating further optimization of NN architectures and interpretability improvements.

## 3 Method

In this section, we detail the generalization component proposed to mitigate cross-structural performance losses in DD. An overview of our method is illustrated in Figure 3.

### 3.1 Preliminaries

Suppose there is an original dataset $T=\{(x_{1},y_{1}),\dots,(x_{|T|},y_{|T|})\}$ with $|T|$ pairs of training samples $x_{t}$ and corresponding labels $y_{t}$. The goal of DD is to synthesize a dataset $S=\{(\hat{x}_{1},\hat{y}_{1}),\dots,(\hat{x}_{|S|},\hat{y}_{|S|})\}$ where $|S|\ll|T|$. The model trained on $S$ is expected to perform similarly to one trained on $T$. For a set of different NN architectures $\theta=\{\vartheta_{1},\dots,\vartheta_{|M|}\}$, we define the performance loss when a distilled dataset $S(\vartheta_{u})$ distilled on $\vartheta_{u}$ is used to train $\vartheta_{v}$ as $\Delta\text{Acc}(\vartheta_{v}|\vartheta_{u})=\text{Acc}_{S}(\vartheta_{v})^{\vartheta_{v}}-\text{Acc}_{S}(\vartheta_{u})^{\vartheta_{v}}$. $\text{Acc}_{S}(\vartheta_{v})^{\vartheta_{v}}$ denotes the accuracy obtained by distilling and training on the same model architecture $\vartheta_{v}$. Our objective is to minimize the total cross-structural performance loss for all $\vartheta_{u},\vartheta_{v}\in\theta$: $\min\sum_{i=1}^{|M|}\sum_{j\neq i}^{|M|}\Delta\text{Acc}(\vartheta_{v}|\vartheta_{u})$.

### 3.2 Heterogeneous and meta features

For data’s heterogeneous and meta features’ visual presence, we initially use Grad-CAM to capture the CAM ${C_{s}}=\bigcup\limits_{m}^{\left|M\right|}{\left\{{c_{s}^{m}}\right\}}$ of the data ${{\hat{x}}_{s}}$ across different pre-trained model architectures $\theta$. We then interpolate the CAMs of the data to the same size, and each matrix element of all CAMs is normalized to a range from 0 to 1. The processed CAMs is ${{\tilde{C}}_{s}}=\bigcup\limits_{m}^{\left|M\right|}{\left\{{\tilde{c}_{s}^{m}}\right\}}$. To mitigate the influence of random factors, we disregard the low-confidence activation regions within the CAMs, specifically those areas where the values are below 0.5. For high-confidence CAMs, we define the pixel locations of the meta features as the overlapping sections across all NNs’ CAMs. The mask representing the meta feature is denoted as :

$$ ${\mu_{s}}=\prod\limits_{m}^{\left|M\right|}{H\left({\tilde{c}_{s}^{m}}\right)},H\left(c\right)_{i,j}=\begin{cases}1&\text{if }c_{i,j}\geq 0.5\\ 0&\text{otherwise}\end{cases}$ (1) $$

Subtracting the common feature areas from each CAM yields each architecture’s heterogeneous areas of focus. The mask of heterogeneous feature areas of the NN architecture $\vartheta_{u}$ in the distilled image $x_{s}$ is represented as:

$$ $\beta_{s}^{m}=H\left({\tilde{c}_{s}^{m}}\right)-{\mu_{s}}$ (2) $$

Figure: Figure 4: ViT’s validation accuracy on different erased TinyImagenet. The numbers represent the difference in accuracy between the erased and the original dataset.
Refer to caption: x5.png

Further experiments using TinyImagenet confirm that our defined heterogeneous and meta features respectively exhibit specific and common preferences across different NN architectures. We initially erase the heterogeneous feature pixels corresponding to different pre-trained NN architectures in TinyImagenet. Architectures are ResNet34, MobileNetV2, GoogleNet, VGG19, EfficientNet, and ViT. Then we train ViT from scratch with the erased TinyImagenet. The accuracy differenece in Figure 4 indicates that ViT suffers the most when losing self-heterogeneous features and the least loss when losing other architectures’ heterogeneous features. We perform the same experiment by erasing the meta features of all architectures in TinyImagenet. Result shows ViT experience significant performance drops after losing meta features. These experiments thoroughly illustrate the distinct natures of heterogeneous and meta features and lay the empirical foundation for our method.

Figure: (a) Distilled ILSVRC-2012 with Sre2L or Sre2L+MetaCAM.
Refer to caption: fig/duibisre2l.png

### 3.3 MetaDD

MetaDD define the NN used in the original DD method as the backbone NN $\vartheta_{\mu}$. To obtain a distilled dataset that generalizes across architectures, MetaDD incorporates pre-trained NNs of various other architectures, referred to as auxiliary networks:

$$ $\theta=\{\vartheta_{1},\dots,\vartheta_{m},\dots,\vartheta_{|M|}\}$ (3) $$

MetaDD employs an architecture-invariant loss function to backpropagate and obtain CAMs on distilled data for different auxiliary networks and then normalizes these CAMs. By reducing the variance at the same locations across these CAMs, the CAMs tend to be similar. Through multiple rounds of updates, MetaDD enhances the test performance of the distilled dataset on other NNs.

Architecture-invariant Loss. We use a mixed loss function of cross-entropy and KL divergence to obtain the Grad-CAM images, specifically expressed as:

$$ ${L_{ai}}=\sum\nolimits_{m}^{\left|M\right|}{{L_{ce}}\left({{\vartheta_{m}}\left(x\right),y}\right)}+\sum\nolimits_{m}^{\left|M\right|}{{L_{kd}}\left({{\vartheta_{m}}\left(x\right),{\vartheta_{\mu}}\left(x\right)}\right)}$ (4) $$

$$ ${L_{kd}}\left({{\vartheta_{m}}\left(x\right),{\vartheta_{\mu}}\left(x\right)}\right)={\vartheta_{m}}\left(x\right)\log\frac{{{\vartheta_{m}}\left(x\right)}}{{{\vartheta_{\mu}}\left(x\right)}}$ (5) $$

Compared to solely utilizing cross-entropy loss for backpropagation to obtain CAMs from auxiliary NNs and aligning these CAMs, the architecture-invariant Loss, which includes an additional KL divergence loss, offers significant advantages:
the CAMs generated by architecture-invariant loss reflect the distilled data features that need to be focused on when transferring knowledge from the auxiliary NNs to the main NN. Consequently, architecture-invariant loss maximumly displays heterogeneous features antagonistic to the main NN, which will be transferred to meta features.

Figure: Algorithm 1 MetaDD Algorithm

Modified Class Active Map. We utilize a modified Grad-CAM [] to obtain activation maps from various convolutional NNs. We initially perform a forward pass to acquire the unflattened feature maps $A$ from the last fully connected layer. Let $A^{k}$ represent the feature map activations of the $k$-th channel for $A$. Then MetaDD computes the gradient of $L_{ai}$, concerning feature map activations $A^{k}$. These gradients flowing back are global-average-pooled to obtain the neuron importance weights $\alpha_{k}^{c}$. Then, the linear combination of these weighted activation maps gives the class-discriminative localization map $c$ used to highlight the important regions.

$$ $\alpha_{k}=\frac{1}{I*J}\sum_{i}^{I}\sum_{j}^{J}\frac{\partial L_{ai}}{\partial A_{ij}^{k}},\quad c=\left(\sum_{k}\alpha_{k}A^{k}\right)$ (6) $$

where $i$ and $j$ are the spatial dimensions of the feature map, and $Z$ is the total number of elements in the feature map.

For Vision Transformers, we consider the output of the last transformer layer $\mathbf{A}\in\mathbb{R}^{N\times D}$, where $N$ is the number of patches and $D$ is the dimension of features per patch. The class token’s output $\mathbf{Z}$ is utilized by an MLP head to generate class predictions $y^{c}$. Attention scores are computed as:

$$ $W_{\text{agg}}=\sum_{\text{heads}}W_{\text{head}}[\text{cls},:]$ (7) $$

where $\mathbf{Q}$ and $\mathbf{K}$ are the query and key matrices from the multi-head self-attention mechanism.
The CAM is generated by:

$$ $W_{\text{agg}}=\sum_{\text{heads}}W_{\text{head}}[\text{cls},:],\quad c=\left(W_{\text{agg}}\cdot\mathbf{Z}\right)$ (8) $$

where $W_{\text{agg}}$ is the aggregated attention across heads. More different from Grad-CAM, we do not employ the ReLU function in Equation 6 and 8. This is because the negative parts of the activation maps are also essential for our optimization. Meanwhile, we ensure that the positive values in all different CAMs are maximized as much as possible:

$$ $L_{pos}=\sum\nolimits_{\rm{i}}{\sum\nolimits_{j}{{c_{i,j}}}}$ (9) $$

CAM Variance Loss. After obtaining the heterogeneous CAMs of all auxiliary networks relative to the backbone network, we interpolate and normalize these CAMs to the same size and range:

$$ $\tilde{c}_{s}^{m}=\frac{{P\left({c_{s}^{m}}\right)-\min P\left({c_{s}^{m}}\right)}}{{\max P\left({c_{s}^{m}}\right)-\min P\left({c_{s}^{m}}\right)}}$ (10) $$

$P\left(\right)$ is the interpolating operation. We then calculate the variance at the same positions in the processed heterogeneous CAMs:

$$ ${\mathop{\rm var}}\left({{{\tilde{c}}_{s}}}\right)=\frac{1}{I*J}\sum\nolimits_{i}^{I}{\sum\nolimits_{j}^{J}{{{\left({\frac{1}{M}\sum\nolimits_{m}^{\left|M\right|}{\tilde{c}_{s,i,j}^{m}-\tilde{c}_{s,i,j}^{m}}}\right)}^{2}}}}$ (11) $$

By minimizing the variance across all positions, the heterogeneous CAMs will tend to be similar.
In the process of becoming similar, the features of the data distilled by the backbone NN will be more acceptable to other network architectures. The final loss function of MetaDD is:

$$ $L_{all}(x_{s})=L_{dd}+L_{ai}+\mathop{\rm var}\left({{{\tilde{c}}_{s}}}\right)+L_{pos}$ (12) $$

$L_{dd}$ is DD method loss function. Through MetaDD, we ensure that the distilled data features are as universal as possible rather than heterogeneous. The process of MetaDD is shown in Algorithm 1.

During the process of obtaining heterogeneous CAMs, the parameters of all different pre-trained NN architectures are frozen. Using pre-trained models with frozen parameters implies low VRAM consumption. Thus, while encompassing multiple different pre-trained NNs, MetaDD still saves computational resources. As a low computational consumption component, MetaDD can be combined with various DD methods to achieve optimal cross-structural training generalization.

## 4 Experiments

**Table 1: The cross-architecture generalization experiments on ILSVRC-2012 and Tiny-ImageNet. $L_{ai}$ is DD using architecture-invariant loss function without generating CAMs.**
| ILSVRC-2012 (IPC = 10) |  |  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Method | Component | Auxiliary/Seen | Unseen | Average |  |  |  |  |  |  |
| ResNet34 | MobileNetV2 | GoogleNet | ViT-B-16 | AlexNet | ResNet50 | Vgg19 | Swin-S |  |  |  |
| TesLa | none | 11.8$\pm$1.3 | 9.6$\pm$1.1 | 10.8$\pm$0.6 | 11.2$\pm$1.7 | 9.2$\pm$1.2 | 11.7$\pm$0.6 | 10.8$\pm$0.9 | 10.3$\pm$0.7 | 10.6 |
| ModelPool | 12.1$\pm$1.1 | 9.9$\pm$1.2 | 10.9$\pm$0.3 | 11.4$\pm$1.2 | 9.6$\pm$0.7 | 11.4$\pm$0.3 | 10.3$\pm$0.7 | 10.6$\pm$0.4 | 10.8 |  |
| GLaD | 12.8$\pm$1.1 | 11.1$\pm$0.6 | 11.9$\pm$1.1 | 12.1$\pm$0.3 | 11.7$\pm$1.2 | 12.4$\pm$1.3 | 12.9$\pm$0.6 | 11.7$\pm$1.1 | 12.1 |  |
| MetaDD | 13.1$\pm$0.3 | 13.4$\pm$0.2 | 14.2$\pm$0.3 | 12.9$\pm$0.6 | 12.4$\pm$0.2 | 13.2$\pm$0.1 | 13.7$\pm$0.2 | 11.9$\pm$0.5 | 13.1 |  |
| Sre2L | none | 13.3$\pm$0.1 | 12.1$\pm$0.3 | 12.7$\pm$0.3 | 13.7$\pm$0.3 | 12.9$\pm$0.8 | 11.8$\pm$0.2 | 11.9$\pm$0.2 | 13.1$\pm$0.6 | 12.9 |
| ModelPool | 13.5$\pm$0.7 | 12.3$\pm$0.5 | 12.9$\pm$0.3 | 13.7$\pm$0.5 | 13.2$\pm$0.7 | 12.2$\pm$0.3 | 12.2$\pm$0.3 | 13.3$\pm$0.5 | 13.2 |  |
| GLaD | 14.6$\pm$0.2 | 13.8$\pm$0.2 | 14.2$\pm$0.2 | 14.6$\pm$1.2 | 13.6$\pm$0.2 | 12.9$\pm$1.2 | 13.9$\pm$1.2 | 14.2$\pm$0.3 | 13.9 |  |
| MetaDD | 14.8$\pm$0.3 | 14.9$\pm$0.1 | 13.8$\pm$0.6 | 14.2$\pm$0.4 | 14.8$\pm$0.4 | 13.9$\pm$0.2 | 15.8$\pm$0.7 | 15.9$\pm$0.1 | 14.6 |  |

**Table 2: CIFAR10 cross-architecture average accuracy.**
| CIFAR10 |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- |
| Method | Component | Auxiliary/Seen (Average) | Unseen (Average) |  |  |
| IPC = 1 | IPC = 10 | IPC = 1 | IPC = 10 |  |  |
| DC | none | 17.6$\pm$1.1 | 38.1$\pm$0.3 | 16.1$\pm$1.2 | 39.7$\pm$1.1 |
| ModelPool | 17.8$\pm$1.0 | 38.5$\pm$0.7 | 16.5$\pm$1.1 | 39.9$\pm$1.0 |  |
| GLaD | 21.2$\pm$0.4 | 39.1$\pm$1.2 | 20.9$\pm$1.2 | 39.8$\pm$1.2 |  |
| MetaCAM | 22.1$\pm$1.1 | 42.2$\pm$1.1 | 21.3$\pm$1.2 | 40.3$\pm$0.6 |  |
| DM | none | 18.9$\pm$1.2 | 40.1$\pm$1.4 | 17.8$\pm$0.8 | 39.8$\pm$0.6 |
| ModelPool | 18.9$\pm$0.9 | 40.6$\pm$1.1 | 17.9$\pm$0.7 | 40.3$\pm$0.4 |  |
| GLaD | 19.2$\pm$0.3 | 41.2$\pm$0.5 | 18.9$\pm$1.2 | 40.1$\pm$1.2 |  |
| MetaCAM | 20.1$\pm$1.2 | 42.3$\pm$0.7 | 19.2$\pm$0.7 | 40.3$\pm$1.2 |  |
| MTT | none | 37.2$\pm$1.2 | 52.1$\pm$2.1 | 36.2$\pm$0.4 | 50.9$\pm$0.2 |
| ModelPool | 37.6$\pm$1.0 | 52.3$\pm$2.2 | 36.7$\pm$0.1 | 51.2$\pm$0.4 |  |
| GLaD | 38.7$\pm$0.2 | 52.2$\pm$1.2 | 37.8$\pm$0.3 | 51.4$\pm$1.2 |  |
| MetaCAM | 37.9$\pm$1.2 | 53.1$\pm$1.3 | 37.2$\pm$0.2 | 52.2$\pm$0.6 |  |
| Sre2L | none | 41.2$\pm$1.4 | 59.8$\pm$0.3 | 40.1$\pm$1.2 | 58.8$\pm$0.5 |
| ModelPool | 41.5$\pm$1.2 | 60.2$\pm$0.7 | 40.4$\pm$1.0 | 59.1$\pm$0.3 |  |
| GLaD | 42.1$\pm$1.2 | 60.2$\pm$1.1 | 42.8$\pm$1.7 | 59.7$\pm$1.6 |  |
| MetaCAM | 42.5$\pm$1.0 | 60.4$\pm$0.9 | 44.3$\pm$0.8 | 61.2$\pm$1.2 |  |

### 4.1 Experimental Setup

We evaluated our method (MetaDD) for DD from CIFAR-10 at a resolution of 32 x 32, Tiny-ImageNet [] at 64 x 64, and ILSVRC-2012 [] at 224 x 224. Our experimental code is based on open-source repositories for DC, DM, Tesla, MTT, and Sre2L. Tesla represents a memory-optimized version of MTT. For each method, we directly integrated MetaDD into the existing codebases. While keeping the hyperparameters of the existing methods unchanged. We show part of distilled images in Figure 5.

Baselines. In addition to MetaDD, we also report the performance of the existing component GLaD [] and ModelPool []. GLaD stores distilled data as feature vectors and uses a generator to create high-definition images as inputs during NN training. ModelPool randomly selects different architectures from a pool of architectures for DD.

Figure: (a) Distilled ILSVRC-2012 with Sre2L
Refer to caption: x6.png

Neural Architecture. We employed the ConvNet [] architecture as our backbone NN for DC, DM, and MTT/Tesla. The Depth-n ConvNet consists of n blocks followed by a fully connected layer.
Each block comprises a 3x3 convolutional layer with 128 filters, instance normalization[58], ReLU nonlinearity, and a 2x2 average pooling with a stride of 2.
For Sre2L, we use ResNet18 as our backbone NN. we use ResNet34 [], MobileNetV2 [], GoogleNet [], and ViT-B-16 [] as our auxiliary NN architectures. All auxiliary NNs are pre-trained using original datasets. We trained the distilled dataset using 8 different NN architectures to test the algorithm’s cross-architecture generalizability. In addition to the auxiliary NN architectures, the test also included four architectures not involved in DD: AlexNet [], ResNet50, Vgg19 [], and Swin-S [].

Evaluation Metrics. We evaluated the cross-architecture generalizability of the algorithm by averaging the top-1 accuracy of NNs trained on the distilled dataset on the validation set.
This average accuracy measure is a robust indicator of the algorithm’s cross-architecture generalizability.

Training Paradigm. The training paradigm for all NNs and datasets is consistent: it includes Stochastic Gradient Descent (SGD) with 0.9 momentum, $1e-4$ weight decay, followed by 500 rounds of linear warm-up and then 500 rounds of cosine decay. Each architecture employs an appropriate (fixed) initial learning rate.
The training process is repeated three times, and the average validation accuracy $\pm$ one standard deviation is reported.

### 4.2 Cross-Architecture Generalization

We initially validated our algorithm’s capacity for enhancing cross-architecture generalization at ILSVRC-2012 and Tiny-ImageNet.
In Table 1, we employed GLaD, ModelPool, and MetaDD to assist DD methods. The results from Table 1 demonstrate that MetaDD effectively reduces overfitting in the backbone NNs.
Compared to other baselines, our method generally outperforms in most cases. Moreover, NN architectures included in the auxiliary NN set show similar performance to the unseen NN architectures.
Hence, by incorporating specified NN architectures to MetaDD, MetaDD can offer customized services tailored to situations with a specific focus on different NN architectures.
Following this, in Table 2, we distilled datasets of varying scales under CIFAR10 and conducted analogous experiments. The results in Table 2 indicate that our method still helps mitigate overfitting. Compared to distillation at higher resolutions, GLaD exhibits weaker performance.

**Table 3: Memory cost and training time for different methods on CIFAR10 and ILSVRC-2012 datasets with ipc=10.**
| Method(Dataset) | Component | Memory(GB) | Time (Minutes) |
| --- | --- | --- | --- |
| MTT(CIFAR10) | - | 19.9 | 98 |
| MetaCAM | 22.6 | 107 |  |
| GlaD | 39.1 | 152 |  |
| ModelPool | 32.4 | 219 |  |
| TesLa(ILSVRC-2012) | - | 68.7 | 1538 |
| MetaCAM | 76.4 | 1601 |  |
| GlaD | 119.1 | 1912 |  |
| ModelPool | 89.4 | 2125 |  |

### 4.3 Meta Features’ Contagious Generalizability

In the previous section, we demonstrated that MetaDD can enhance performance on NN architectures not included in the auxiliary NNs (unseen NNs). This phenomenon, which we term ”contagious generalizability”, is attributed to the universality of MetaDD. Figure 6 illustrates the meta features of distilled data across both unseen and seen architectures. Images generated by the original DD method exhibit almost no meta features. However, with the integration of MetaDD, the distilled images possess meta features recognizable by seen NNs and unseen NNs. This indicates that the meta features introduced by MetaDD are typical and widely applicable, further substantiating the efficacy of MetaDD in enhancing cross-architecture generalizability.

### 4.4 Training Cost Analysis

We compare the GPU memory consumption of our method with that of GLaD. We kept all other conditions identical between the two methods. As shown in Table 3, on CIFAR-10, our method reduces memory usage by 2̃$x$ compared to GLaD. The runtime is reduced by 0̃.3$x$. GLaD consumes a significant amount of memory due to the use of generators, whereas our method maintains low memory usage even while accommodating 4 auxiliary NNs. Because the parameters of the auxiliary NNs are always kept frozen, we can scale to a larger number of auxiliary NNs.

### 4.5 Ablation Study

**Table 4: Ablation Study on CIFAR10 with IPC=10.**
| Method | Loss | Accuracy |
| --- | --- | --- |
| MTT | None | $52.1\pm 2.1$ |
| with $L_{ai}$ | $52.4\pm 1.3$ |  |
| with $L_{pos}$ | $52.3\pm 0.7$ |  |
| with $\mathop{\rm var}\left({{{\tilde{c}}_{s}}}\right)$ | $52.9\pm 0.5$ |  |
| Sre2L | None | $59.8\pm 0.3$ |
| with $L_{ai}$ | $59.9\pm 1.1$ |  |
| with $L_{pos}$ | $60.2\pm 0.2$ |  |
| with $\mathop{\rm var}\left({{{\tilde{c}}_{s}}}\right)$ | $60.4\pm 0.2$ |  |

We distilled CIFAR-10 by adding different loss function components. From the experimental results in Table 4, it can be seen that $\mathop{\rm var}\left({{{\tilde{c}}_{s}}}\right)$ has the greatest effect, while $L_{ai}$, $L_{pos}$ have the secondary effect. The benefit of $\mathop{\rm var}\left({{{\tilde{c}}_{s}}}\right)$ comes from obtaining consistent features recognized by different architectures, $L_{pos}$ merely makes the CAM features more visible, and $L_{ai}$ enable the architecture to benefit from knowledge transferring.

## 5 Conclusion

We introduce MetaDD, a new component specifically designed to enhance the cross-architecture generalizability of DD. MetaDD delivers the dual advantages of minimal additional computational overhead and improved performance. By delving into the factors that limit cross-architecture generalizability, MetaDD uncovers the unique feature recognition mechanisms inherent to different neural network architectures, which often prioritize diverse and heterogeneous features. However, these architectures also adhere to certain shared aesthetic or structural standards. MetaDD enhances cross-architecture generalizability by amplifying the representation of meta features that align with these shared standards. It achieves this by synthesizing meta features through the integration of unified CAM outputs from various neural networks, ensuring these meta features are broadly recognized and effectively utilized across different architectures.

## Peer Reviews

confidence: 4
contribution: 4
initial_rating: 8
presentation: 3
questions: 1. Why is the Dream algorithm used for comparison with MetaDD, despite not being designed for cross-architecture generalization? Would a comparison with more relevant algorithms be more suitable?

2. Could the roles of each component in the loss function be explained in more detail? How does each component contribute to the overall optimization process?

3. Why does an increase in meta-features result in a reduction of heterogeneous features? Could additional theoretical or empirical support clarify this relationship?
soundness: 3
strengths: - This paper introduces **MetaDD**, a framework designed to significantly improve dataset distillation (DD) by addressing the challenge of cross-architecture generalization, which remains a critical limitation in existing DD methods. By enabling distilled datasets to generalize effectively across various neural network architectures, MetaDD offers a promising solution to one of the primary bottlenecks in DD research.

- **MetaDD’s key innovation** lies in its integration of class activation maps (CAM) into DD algorithms, providing a novel approach to analyzing and understanding different types of features in synthetic datasets. This integration not only enhances the interpretability of distilled data but also strengthens generalizability by aligning feature distributions across architectures, enabling the framework to improve model performance on a variety of architectures.

- **Flexible Integration**: MetaDD is designed as a modular, adaptable component that can be seamlessly incorporated into a range of DD algorithms, enhancing their effectiveness without requiring substantial modifications to the underlying architecture. This flexibility makes MetaDD highly versatile and applicable to diverse DD applications, promoting broader adoption across different neural network models.

- **Technical Rigor**: The framework is supported by robust empirical evidence and thorough analysis, demonstrating its soundness and reliability. Each claim is backed by theoretical underpinnings and practical results, reinforcing MetaDD’s credibility as an effective solution for enhancing DD performance.

- **Performance and Transferability**: Experimental findings show that MetaDD significantly improves the transferability of distilled data, maintaining high performance even on previously unseen architectures. This improved transferability marks a step forward for DD methods, extending their usability and robustness in real-world applications where unseen architectures are common.

- **Clear Structure and Reproducibility**: The paper is well-organized and provides detailed methodology, ensuring that expert readers can easily reproduce the results. This clarity promotes transparency and encourages further exploration by other researchers in the field.

- **Contribution to the Field**: MetaDD advances the state of the art in dataset distillation by effectively increasing the representation of meta-features while reducing heterogeneous features in distilled data, creating a more balanced and efficient feature set for model training. This balance enhances the utility of distilled datasets and lays the groundwork for further improvements in generalization.

- **Balanced Assessment**: The authors present an honest and well-rounded view of MetaDD’s capabilities, highlighting both its strengths and areas for potential improvement. This balanced perspective not only builds trust in the research but also offers valuable insights for future work, positioning MetaDD as a strong foundation for advancing dataset distillation techniques.
summary: This paper introduces the MetaDD framework to improve cross-architecture generalization in dataset distillation (DD) methods. Synthetic datasets generated from specific NN architectures show more heterogeneous features and fewer meta-features, reducing cross-architecture performance. MetaDD addresses this with an architecture-invariant loss, aligning and diversifying GradCAM outputs to optimize feature distribution. Using cross-entropy and Kullback-Leibler divergence terms, this approach enhances diversity and consistency across networks. Experiments show MetaDD outperforms current methods, validating its effectiveness in boosting meta-feature recognition and cross-architecture consistency.
weaknesses: - **Inappropriate Comparison Selection**: The paper compares the Dream algorithm with MetaDD, but since Dream is not designed to address cross-architecture generalization, this comparison lacks a sound basis. More persuasive comparisons could be made with algorithms that share similar objectives.

- **Inadequate Explanation of Loss Function Components**: The individual roles of each component within the loss function are not thoroughly explained, resulting in an unclear presentation of the design rationale. It would be beneficial to detail the contribution of each component to the optimization process, clarifying the method's effectiveness.

- **Unclear Relationship Between Meta-Features and Heterogeneous Features**: The paper does not clearly explain why an increase in meta-features would result in a reduction of heterogeneous features. Additional theoretical or empirical evidence on this relationship would strengthen the method's scientific validity and coherence.

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 1
questions: - The authors state that “MetaDD increases meta features and reduces heterogeneous features in distilled data.” Could the authors provide quantitative data on the proportion of area occupied by meta features to substantiate this claim?
- In line 314, “Q and K are the query and key matrices.” However, the reviewer notes that equation (7) does not include Q and K but instead utilizes W_{head}. Could the authors elaborate?
- The column name in Table 5. Does the column “component” means “the number of auxiliary models” and “Quantity” means “method”?
- In Equation (9), should there be a summation over $m$ in the loss function?
soundness: 2
strengths: The analysis of meta and heterogeneous features provides strong motivation for this study. The approach of enhancing cross-architecture generalization is innovative and promising.
summary: This work, titled MetaDD, introduces a distillation plugin module designed to enhance the cross-architecture generalizability of distilled datasets. MetaDD promotes the universality of distilled image features across varying architectural frameworks.
weaknesses: The improvements brought by MetaDD appear modest. 

Numerous typographical errors and incorrect formulas are present, and the authors should refine the manuscript for clarity.
Specific issues include:
- Typographical errors in lines 158 and 159: “improve the vision transformer CAMZhu et al. (2023);…”
- Incorrect use of subindices i, j, u, and v in line 176.
- In Table 1(a), *Sre2L + MetaDD* performs worse than *Sre2L+GLaD* on GoogleNet and ViT-B-16; however, *Sre2L + MetaDD* is incorrectly highlighted in bold.
- In lines 478 to 480, “our method reduces memory usage by $2x$ compared to GLaD”. The author should use $2 \times$ instead of $2x$

---

confidence: 4
contribution: 2
initial_rating: 3
presentation: 1
questions: - Is the bold formatting in the tables somewhat inappropriate? The best results under the same experimental setup should be bolded, rather than just the results of your own method.
- How can the appropriate combination of auxiliary networks be effectively selected in practical applications?
- MetaDD introduces architecture-invariant loss functions and feature alignment techniques to maximize meta features. Has there been a detailed analysis of the weights of these loss terms and their impact on the final generalization performance?
soundness: 2
strengths: - MetaDD is characterized by low memory consumption and can be seamlessly integrated with existing dataset distillation methods.
- It introduces a certain level of innovation by incorporating architecture-invariant loss functions and feature alignment techniques, addressing the issue of poor generalization performance in dataset distillation across different neural network architectures.
- Class Activation Maps (CAMs) are used to visually demonstrate the feature preferences of different neural network architectures, making it more intuitive.
summary: MetaDD offers a fresh approach to improving dataset distillation by enhancing generalizability across different architectures. By introducing architecture-invariant loss and concentrating on increasing meta features, it achieves notable advancements over current methods, delivering better performance and reducing memory consumption across multiple neural network architectures.
weaknesses: - The experimental results are not very optimistic, as the improvement in several groups is negligible.
- Multiple different auxiliary neural networks were added to enhance generalization performance, but the paper does not provide a detailed discussion on how to select the appropriate combination of auxiliary networks.
- When handling large datasets, whether the introduction of multiple auxiliary networks and architecture-invariant loss functions will significantly impact training time remains unclear.

---

confidence: 4
contribution: 2
initial_rating: 3
presentation: 1
questions: In Fig1, shouldn't the CAM visualizations for cross-arch and same-arch be identical? Since the original dataset hasn't undergone the DD process and is directly tested on a single architecture, there shouldn’t be a distinction between different architectures.

The table of hyperparameters does not clarify the proportions of the individual losses. Are they set to a 1:1:1 ratio? This could potentially diminish the optimization benefits originally intended in the DD process.

There are labeling errors, such as in Table 1(a) under SRe2L seen, where MetaDD and GLaD are listed, but GLaD should be bolded.
Table 2 does not specify which network was used for the experiments, and Table 4 fails to indicate the type of GPU used for running the timing experiments.

A possible issue is that the paper exceeds the 10-page limit.

The layout of various figures and tables is not aesthetically pleasing.
soundness: 2
strengths: The exploration of cross-architecture generalization, which has been rarely addressed in previous works, opens up a new direction in the field.

Despite introducing multiple auxiliary networks, the method maintains memory efficiency, making it a notable contribution. 

MetaDD can also be seamlessly integrated into existing dataset distillation techniques without altering their core mechanisms.

Furthermore, the use of Class Activation Maps (CAMs) to visualize and analyze the features responsible for cross-architecture generalization enhances the interpretability of the approach, adding significant value to the paper.
summary: This paper primarily focuses on enhancing the generalization performance of domain decomposition (DD) across different neural network architectures. Specifically, the authors decompose features into meta features and heterogeneous features, aiming to maximize the proportion of meta features while minimizing the contribution of heterogeneous features. This process is achieved by adjusting the activation function accordingly. Moreover, to reduce memory consumption, the method involves persistently freezing various layers in the neural network architectures during the DD process. As a result, the approach demonstrates significant improvements in performance.
weaknesses: The performance improvements are marginal, and in some cases, they can be attributed to variance rather than significant gains. The ablation study shows a similar trend—introducing $L_{ai}$ yields improvements that could be due to errors, particularly in the SRe2L method.

This reliance on pre-trained NNs could limit its flexibility.

The number of experiments conducted on different datasets is somewhat limited. It would be beneficial to validate the model's performance across a broader range of datasets to strengthen the conclusions.

Additionally, the ablation study is not sufficiently comprehensive. It may be helpful to explore different combinations of losses in the ablation study to provide deeper insights into their individual and collective impacts.