Audio-Visual Dataset Distillation
Authors: Saksham Singh Kushwaha
Siva Sai Nagender Vasireddy
Kai Wang
Yapeng Tian
Venue: Accepted by TMLR ()

Abstract: In this article, we introduce \textit{audio-visual dataset distillation}, a task to construct a smaller yet representative synthetic audio-visual dataset that maintains the cross-modal semantic association between audio and visual modalities. Dataset distillation techniques have primarily focused on image classification. However, with the growing capabilities of audio-visual models and the vast datasets required for their training, it is necessary to explore distillation methods beyond the visual modality. Our approach builds upon the foundation of Distribution Matching (DM), extending it to handle the unique challenges of audio-visual data. A key challenge is to jointly learn synthetic data that distills both the modality-wise information and natural alignment from real audio-visual data. We introduce a vanilla audio-visual distribution matching framework that separately trains visual-only and audio-only DM components, enabling us to investigate the effectiveness of audio-visual integration and various multimodal fusion methods. To address the limitations of unimodal distillation, we propose two novel matching losses: implicit cross-matching and cross-modal gap matching. These losses work in conjunction with the vanilla unimodal distribution matching loss to enforce cross-modal alignment and enhance the audio-visual dataset distillation process. Extensive audio-visual classification and retrieval experiments on four audio-visual datasets, AVE, MUSIC-21, VGGSound, and VGGSound-10K, demonstrate the effectiveness of our proposed matching approaches and validate the benefits of audio-visual integration with condensed data. This work establishes a new frontier in audio-visual dataset distillation, paving the way for further advancements in this exciting field. \textit{Our source code and pre-trained models will be released}.

Published in Transactions on Machine Learning Research (10/2024)
Madio-Vissul Dataset Distillation
Saksham Singh Kushwaha
Department of Computer Science The University of Texas at Dallas
Siva Sai Nagender Vasireddy
irteaainageedderr.nsvviedddddddd edu
Deportment of Computer Science The University of Teras at Dallas
Kai Wang
kai, woo..uonn@@comm.muus..d www.:mpppp.............
Institutes of Data Science & School of Computing National University of Singapore
Yapeng Tian
yapeng Bineennglaa@@@ualllla...ao
Department of Computer Science The University of Tezas at Dallas
Reviewed on OpenReview: https: // opt.///eeep...ee...mmmm net/ ht///forur forum? id= IJtbuSXXm
Abstract
In this article, introduce audio-visual dataset distillation, task to construct smaller yet representative synthetic audio-visual dataset that maintains the cross-modal seman- tic association between audio and visual modalities. Dataset distillation techniques have primarily focused on image classification. However, with the growing capabilities of audio- visual models and the vast datasets required for their training, is necessary to explore distillaton methods beyond the visual modality. Our approach builds upon the foundation of Distribution Matching (DM), extending to handle the unique challenges of audio-visual data. A key challenge is to jointly learn synthetic data that distills both the modality- wise information and natural aligument. from real audio-visual data. We introduce vanilla. audio-visual distribution matching framework that separately trains visual-only and audio- only DM components, cnabling us to investigate the effectiveness of andio-visual integration and various multimodal fusion methods. To address the limitations of unimodal distilla- tion, we propose two novel matching losses: implicit cross-matching and cross-modal gap matching. These losses work in conjunction with the vanilla unimodal distribution match- ing loss enforce cross-modal alignment and enhance the andio-visual dataset distillation process. Extensive audio-visual classification and retrieval experiments on four audio-visual datasets, AVE, MUSIC-21, VGGSound, and VGGSound-10K, demonstrate the effectiveness of our proposed matching approaches and validate the benefits of audio-visual integration with condensed data. This work establishes a new frontier in audio-visual dataset distil- lation, paving the way for further advancements this exciting field. Source code and pre-trained model: https://githu/: http://////aaassssssag/////a///
1 Introduction
Dataset distillation aims to learn a condensed dataset such that it retains most of the essential information of the entire training data. Recent progress in datasot distillation techniques, such as gradient matching
(Zhao al. 2020 Zhao Bilen 2021), trajectory matching (Cazenavette et al. 2022 Wu al.) 2023 Liu
et al. 2023 and distribution matching (Zhao & Bilen 2023 Zhao et. al. 2023: Wang al. 2022 have

---

Published in Transactions on Machine Learning Research (10/2024)
Real audio-visual
(A)
Synthetic audio-visual data
(B)
Mccurang (%)
wwwwmmm w
visual embedding
IPC=1
IPC=10
IPC=20
spaces
Visual
Audio
Audio-Visusl
sudio embedding spaces
s-syyteeii visual ambadding
*
symthetic audio embedding
real visual embedding
real audio embedding
DM
Ours
Figure 1: Vanilla audio-visual dataset distillation using distribution matching (A) The synthetic audio and visual data are learned by minimizing the distribution discrepancy between real and synthetic data these sampled unimodal embedding spaces. (B) Mean test accuracy plot of VGGS- dataset for different images per class (IPC) settings. We observe that joint audio-visual can improve performance in data distillation. (C) Feature distribution plot of real audio())-visual( and synthetic audio( --iiuul((+)))) data for vanilla distribution matching(DM) and Our approach. We observe that independently distilled audio-visual syn- thetic data in DM is pulled towards their modality centers while our additional joint audio-visual losses better cover the real distribution.
achieved remarkable performance on the image datasets. For example, DATM (Guo et al. 2023 achieves lossless distillation using merely 1/5 and 1 1/10 original sizes of CIFAR100 (Krizhevsky et al. 2209)) and TinylmageNet (Le Yang 2015 respectively. However, their potential in other domains remains largely
underexplored.
With the recent. advancements in audio-visual learning (Zhao et al., 2018 Tian et al. 2018 Lin et al. 2019
Gao et al.] 2018
Wei et al.) 2022), the size of audio-visual datasets (Chen et al., 2020c Xu et al. 2016
Gemmeke el al. 2017) have signilicantly increased, which leads to the heavy storage and computal ional cost of training on these datasets. In this work, we investigate the extension of the dataset distillation to the audio-visual domain. Unlike image distillation (Wang et al. 2018 Cazenavette et al., 2022), audio- visual distillation presents unique challenges: preserving complex cross-mnodal correlations and addressing the complexities of high-resolution images and the additional audio modality.
Andio-visual integration has proven beneficial for various audio-visual tasks, such as audio-visual event Wu et al. 2019 Lin et al. 2019), audio-visual sound separation (Zhao et al. 2018 Gao et al.] 2018 Tian et al. 2021], and audio-visual action recognition (Kazakos et al.] 2019 Zhang et al. 2022 Nagrani et al. 2020 This success raises an important question: Is audio-visual integration still effective when applied to distilled audio and visual data? To explore this, we propose andio-visual dataset distillation, which aims learn smaller, yet representative synthetic audio-visual dataset that useful for audio-visual learning tasks, distilled from the original large datasct. The task is inherently challenging. An effective audio-visual dataset distillation approach not only needs condense data from two different modalities, but also preserve the natural cross-modal association between them to ensure the effectiveness of the synthetic multinodal data.
localization (Tian et al. 2018
To explore this new problem, we use audio-visual event recognition as the main proxy task and build our approach on top of Distribution Matching (DM) (Zhao & Bilen 2023). We first propose a vanilla audio- visual distribution matching approach in which we separately train the visual-only and audio-only DM (as shown in Fig. [ This vamilla approach allows evaluate the effectiveness of audio-visual integration with condensed data and analyze the impact of different multimodal fusion methods audio-visual event
2

---

Published in Transactions on Machine Learning Research (10/2024)
recognition performance. The results are presented in Fig. DB. which demonstrates the advantage of joint audio-visual integration over single audio or visual modality in audio-visual dataset distillation setting.
However, distilling audio and visual modalities independently may not effectively capture the natural inter- play between audio-visual elements during the condensation from real to synthetic data (as shown in Fig. (o). And to address this limitation, we further introduce two novel matching losses: implicit cross-matching and cross-modal gap matching, These losses align distributions between synthetic and real data in joint spaces and work in conjunction with vanilla unimodal distribution matching loss in DM to enhance the audio- visual data distillation process. Their combined effect ensures that the synthetic data closely represents the real data, effectively aligning unimodal, cross-modal, and modality gap distributions, resulting in effective audio-visual event recognition and retrieval with the synthetic data. Furthermore, we improve the synthetic data initialization and utilize the high-resolution images (along with audio modality) using herding- based initialization (Welling) 2009) and factor technique (Kim et al.) 2022), respectively. The herding aligns the initial synthetic data distribution with the real training data, while the factor method increases the number of features helping overcome the redundancy. Our extensive experiments on four widely used audio-visual datasets: AVE (Tian et al., 2018), MUSIC-21 (Zhao et al.) 2019), VGGSound (Chen et al.) (020c)), and VGGS-10k (Chen et al. 2020c) support the following findings: effective joint audio-visual integration out- performs unimodal performance in audio-visual dataset distillation, implicit cross- mass-in and cross-moda gap matching improves the vanilla audio-visual distribution matching by distilling the audio-visual alignment into synthetic data, herding initialization & factor technique further helps improve audio-visual distillation.
The contributions of our work include: 1) a novel audio-visual dataset distillation problem that aims to compress the knowledge of large audio-visual datasets into much smaller synthetic ones. To the best of our knowledge, this is the first work in audio-visual dataset distillation; 2) systematical investigation into audio-visual dataset distillation to evaluate the efficacy of condensed audio-visual data for andio-visual event recognition; 3) two new audio-visual distribution matching losses that align distributions between synthetic and real data in joint spaces, enforcing cross-modal alignment; and 4) extensive experiments on four audio- visual dalasets validating that audio-visual integration with synthetic data is still helpful and our approach can outperform other dataset distillation baselines.
2 Related Work
Audio-Visual Learning. Videos consist of naturally co-occurring audio and visual signals. To exploit the synchronized and complementary information in the two modalities, several audio-visual learning problems
have been explored, such as self-supervised representation learning (Arandjelovic & Zisserman 2017 2018
Aytar et al.) 2016: Lin et al.
2023
Gong ct al. 2022
Owens & Efros
2018 Hu ct al. 2019). audio-visual
sound separation (Zhao el al.
2018
Gan et al. 2020
Gao et al. 2018
Tian et al.] 2021 Zhou et al. 2020).
audio-visual action recognition (Zhang et al. 2022
Lee al., 2020
Nagrani et al. 2020), audio-visus
navigation (Chen et al. 2021 2020a by. visiio event localization (Tian et al.) 2018 Wu et al. 2019:
Lin et al. 2019), etc. Audio-
integration has generally demonstrated the ability to improve model performance compared to unimodal models in these tasks. In this work, we investigate the validity of joint audio-visual integration in the context of dataset distillation. We employ audio-visual event recognition
(Tian et al. 2018 Lin et al. 2023) and mross-modal retrieval (Surís et al. 2018 Kushwaha & Fuentes 2023
Wu et al.)
2023) as proxy to evaluate the effectiveness of condensed data.
Dataset Distillation. The traditional method to reduce the training set size is coreset selection (aatrr et al. 2018 Welling 2009), which heuristically selects subset of training data. A recent approach, dataset distillation condensation aims to learn smaller dataset while still preserving the essential information from the large training datasct. Unlike corcsct selection methods, data condensation methods are not limited by the subset of selected real data. It has been shown to benefit several tasks like efficient neural
architecture search (Zhao & Bilen 2023 Zhao et al. 2020), continual learning (Gu et al. 2023 Sangermano ct al.] 2022; Zhao et al.] 2023). ) foderated learning (Xiong et al. 2023 Huang et al.] 2023). and privacy
preservation (Vinaroz & Park 2023 Dong et al. 2022). The problem was first introduced in (Wagg et al.
2018), which learns condensed data with meta-learning techniques and has since been significantly improved
3

---

Published in Transactions on Machine Learning Research (10/2024)
by more sophisticated techniques like gradient matching Zhao et al. 2020 trajectory matching Cazenavette
et al. 2022), data parameterization (Kim et al. 2022) and feature alignment (Wang et al. 2022 Most of
these methods rely on bi-level optimizations resulting in intensive computation requirements. In contrast, distribution matching (DM) (Zhao & Bilen 2023 avoids this bi-lovel optimization and condenses data by matching the feature distribution of real and synthetic data. Recently, [Wu el. al. (2023) introduced vision-language dataset distillation using joint image-text trajectory matching. However, straightforward extension of their method audio-visual distillation is impractical due to differences in modality, task, and scalability. Unlike text, audio is temporally synchronized with visual components, exhibits greater variability, and is typically represented as spectrograms with rich, isolated time-frequen freeueeyye patterns. Additionally, while they focus on instance-basee retrieval, our task involves class-based tasks. This difference shifts the focus from learning instance-wise distilled data pairs to class-wise pairing in our approach. Furthermore, their ext extension of the MIT approach to instance-win wise distilled data pairs struggles to scale to higher images-per per-class settings in our experiments duc to increased memory requirements. In contrast, wc extend training- raining-basod distillation baselines, moving beyond the coreset selection baselines of of' et al. (2023). In this work, we advance dataset distillation into audio-visual learning, concentrating techniques that effectively condense data from both audio and visual modalities while preserving their cross- modal associations.
3
Method
3.1
Preliminaries
In this section, we describe the problem formulation, introduce the proxy task for evaluating distilled syn- thetic data, and revisit distribution matching for visual-only dataset distillation.
Problem Formulation. Let of and of denote the audio waveform and video frame of the i-th sample, respectively, with 21"h (z xf]) and y the corresponding ground truth category label. Given a large audio-visual training set T= fat", 1 our andio-visual dataset distillation task aims to learn a smaller, yet reprosentative synthetic set S = {st", v/ where st" s!)). This datasct S, with significantly fewer samples |8| < 17,, should encapsulate the essential information contained in T. The ultimate goal is for models trained on each T and S to perform similarly on unseen test data:
E( w>>D [ (x""),v) ~ ((== [e(oos (z*)), v)],
where D is the real test data, l is the loss function (i.e. cross-entropy), de is neural network parameterized by 0, and dæ, and Φeg are networks trained on T and S respectively.
, we use audio-visual event recognition task a proxy to investigate the effectiveness of audio-visual dataset distillation. The task involves predicting the event category of a short video clip, characterized by audio waveform _ and video frame x². We employ an audio-visual network, illustrated in Fig. 2 to integrate data from both modalities. Specifically, the model uses a visual encoder to extract feature f from x" and an audio encoder extracts feature f from the audio spectrogram m transformed from input audio xa. These extracted features are then fused to feature for by a fusion module for predicting the class-event probability p. We utilize cross-entropy loss as the objective function Lo = - ! y; log(p:), where |C] denotes the total number of event categories. Through this task, we will investigate whether integrating synthetic audio- visual data can enhance recognition in dataset distillation and to what extent different fusion methods affect the model performance.
Task. Following image dataset distillation methods (Sachdeva & McAuley 2023 Yu et al., 2023).
Revisit Distribution Matching. Dataset distilla- tion has demonstrated remarkable success in compressing large image datascts into smaller synthetic ones for visual recognition lasks. Distribution matching (DM) [Zhao] & Bilen 2023) stands out as one of the prominent ap- proaches, which aims to generate synthetic data that closely resembles the distribution of real samples in the feature space. This is achieved by minimizing the feature distance between the distributions of real and
Encoder
Fusion
Audio Encoder
Prediced
fa
Figure 2: Audio-visual event recognition.
4

---

Published in Transactions on Machine Learning Research (10/2024)
Vanilla Audio-Visual Distribution Matching
Maacnin OwW
Sp
viaual
encoder
distilled pairs
Rn
Rv
Ra
Sa
synthetic data
Implicit Cross-Matching
R
Ra
S
S
S
Cross-modal Gap Matching
pairs
audio
NNDD
encoder
real data
R₂
Peocneeeaiene Forward pass
R
S
Ra
S
Backpropagation
Figure The proposed audio-visual dataset distillation framework. The synthetic audio-visual feature distribution and real audio-visual feature distribution are matched using three main components: Vanilla audio-visual distribution matching, Implicit Cross- Matching (ICM), and Cross-modal Gap Matching (CGM). The vanilla distribution matching loss ensures the aligmment of the same modality matching while ICM and CGM facilitate cross-modal matching. Ru Ry S., S, are the real audio, real visual, synthetic audio, synthetic visual distribution respectively. Maximum Mean discrepancy (MMD) Gretton et al.) 2012) is used as a measurement for distribution matching. The distilled image and audio spectrogram pairs are shown at an intermediate state and hence slightly modified from the initialized real data.
synthetic samples, ensuring that the synthetic data effectively captures the essential characteristies of the original dataset. Specifically, it uses randomly initialized neural networks as feature extractors and minimizes the spatial distribution using an empirical estimate of maximum mean discrepancy (MMD) (Gretton et al. 2012). DM maps each training image x" E Rd to a lower dimensional space using a family of parametric functions Ve. Rd Re where d' & d. Here, the can be implemented using neural networks with random weights. It also augments data by applying differential Siamese augmentation (+) (Zhao & Bilen 2021 real and synthetic data, where ~ 52 the augmentation parameter. To this end, we will solve the following optimization problem:
mins, E w~n
D
ΣV boo (x,))
S.l j==1 tho (A (s))))
(1)
0
S.
where Pa. is the distribution of network parameters. By minimizing the discrepancy between two distribu- tions various embedding spaces by sampling 0,, we can learn the synthetic visual data S., Upon generating the synthetic dataset S,, we will use as training data train our visual recognition model, optimizing it with the cross-entropy loss. This trained model will then predict class labels for real test image samples.
3.2 Audio- Visual Dataset Distillation
In this section, we will explore dataset distillation approaches capable of simultaneously learning synthetic audio and visual data for audio-visual event recognition. We will first introduce vanilla model that extends visual distribution matching and then present two novel approaches tailored to this task, utilizing joint distribution matching. Our approach is illustrated in Fig. 3
Vanilla Audio- Visual Distribution Matching.
We build our audio-visual dataset distillation approach the top of DM. To tackle the new multinodal task, naive approach to use DM to condense audio

---

Published in Transactions on Machine Learning Research (10/2024)
and visual data separately during data distillation and then combine the trained distilled audio and visual data for audio-visual event recognition. Alongside a visual-only DM loss L detailed in Eq. 1
we
introduce a DM loss for the audio modality:
1 ISa Vo (Aw
base = 11
Σ do. (xx())))
(2)
Taal
S
j=1
where tho denotes randomly initialized audio network. matching optimizes the following objective function:
The vanilla approach to andio-visual distribution
Lbeee
Lasse
+ base
However, distilling modalities separately fails to capture the natural interplay ween real audio-visual data, introduce two joint matching losses to better align the distilled synthetic data.
Distribution Matching Between Joint Spaces. To enforce cross-modal alignment during distribution matching, one straight- forward approach is to introduce a cross-modal distribution match- ing loss. This loss function will minimize the discrepancy between the distributions of real audio data and synthetic visual data, as well between real visual data and synthetic audio data. However, this approach has limitations due to the of randomly initialized models th, and te, for mapping audio and visual data into sep- arate embedding spaces in DM. As shown in Fig. 4 there exists a strong modality gap between the two modalities. Since only the syn- thetic data learnable, real data remains fixed, and model weights are randomiyy initialized and not trainable, directly matching one modality to another cannot mitigate the modality gap and can lead to instability during training (refer to Appendix A for detailed explanation of the failure exploration). This is in contrast to real data, where audio-visual associations naturally exist. To address these challenges, we propose two new learning losses that encour- age cross-modal alignment in a joint embedding space, rather than merely performing cross-modal matching in individual unimodal au- dio and visual spaces.
Figure 4: Feature distribution plot of real audio(=)-visuall and synthetic audio * )visuuaal)()()) data at an inter- mediate training stage of the vanilla audio-visual distribution matching. The audio and visual features are ex- tracted from randomly initialized dio and visual networks, respectively, We can observe the strong modality gap between the two modalities.
Implicit Cross- matshing (ICM). Given the inherent challenges of achieving separate cross-modal distribution matching, we propose a novel approach that bypasses the difficulty by introducing a joint audio-visual distribution matching loss. This loss function effectively aligns the joint audio-visuse distributions between real (D") and synthetic (D*) data, enabling implicit cross-modal distribution matching. The loss is formally defined for each class as follows:
D" = R. R
be (x.))
Me. (A (xx))
(4)
T
D' = 5rh 5
too (Aw +
S. the (Amm
(5)
S.
CiCM = -D11?
(6)
Optimizing this loss term effectively compels our model learn synthetic audio-visual data {Sa, S. that closely resembles and represents the real dataset {Ta, To). Here, the loss term in Eq. 6 can be re-written LieM = [R + R²) (Sa + 22 || - 5²) + (R" 5²) < [| - 5t) ll(Ft 112. This formulation reveals that the loss implicitly enforces cross-modal matching between real audio and synthetic visual data, as well as between real visual and synthetic audio data.
For simplicity, we omit the term in the loss.

---

Published in Transactions on Machine Learning Research (10/2024)
Cross-modes Gap Matching (CGM). Besides the ICM loss, we introduce further constraints to align the distributions of (Ra and S.) and (R, and Sa); as follows:
7
Sul
Da
ve (x,(
Ve. (AW(s)))
(7)
T
|
| vie (A,()))))
D
vo. (Aw (x]))
(8)
|T
[
j=1
MMG |
12
(9)
This addition ensures that the synthetic data closely represent the corresponding real data without misalign-
ing the existing matches of S" ++ R and S + R enforced in unimodal DM and Joint Matching. With a simple re-writing, we can obtain LICM = l| + 5t) (R + S²11 = |I - R²) (S - S2'112 We
can see that it will help to align the modality gap between real and synthetic data to strengthen the joint audio-visual distribution matching.
Final Loss. For training, will jointly optimize the three loss terms:
final
Law AICM LiCM ACGM C
(10)
Here dIeMs and ACGM the weights for implicit cross-matching and cross-modal gap matching losses respectively. These three loss terms work collaboratively to enhance the audio-visual dataset distillation process. Their combined effect ensures that the synthetic data closely corresponds to the real data, aligning unimodal, cross-modal, and modality gap distributions effectively.
Training algorithm. We train the syuthetic data for iterations and for each iteration, we sample random model uhe and tbe for audio and visual embedding. We then randomly sample real audio-visual data batch and audio-visual synthetic data batch and augmentation parameter for each class. We calculate the mean discrepancy between each modality individually and implicit cross- musssmnttching and cross-modal gap-matching representations for each class and then sum them as loss L. We update the synthetic data by backpropagating C for each class with learning rate The overall training algorithm is shown in Appendix Algorithm [
Improved initialization and storage. Herding (Welling 2009) is coreset selection method that greedily selects data points to minimize the coreset center and the original data center. It has shown superior performance among the coreset selection methods in several previous dalaset distillation research (Sachdeva & McAuley 2023 Yu et al. 2023). We adopt herding as it aligns the initial synthetic data distribution with the real data, offering an advantage over random initialization. The factor technique (Zhang et al.) 2023 Kim et al. 2022 Zhao et al.] 2023) aims to increase the number of representative features extracted from synthetic data without any additional cost. Specifically, given a factor parameter each synthetic image is factored into (2 mini-examples and then up-sampled to its original size in training. By combining herding initialization and factor technique, we can further improve our joint audio-visual distribution matching. (More details are provided in Appendix B
4 Experiments
Following the evaluation protocols from previous datasct distillation studies (Zhao & Bilen 2023 Cazenavette et al. 2022). we use audio-visual event. recognition the main proxy task to assess the classification accuracy on held-out test data of deep networks trained from scratch on our distilled audio and visual data.
4.1 Experimental Settings
Datascts. We use four widely used andio-visual datasets to validate distillation mothods. Each sample represents a one-second video clip, comprising center frame and its corresponding audio.
VGGSound & VGGS-10k. VGGSound (Chen et al., [020cc] large scale audio-visual dataset consisting of around 200k YouTube videos from 309 classes. We select the center one-second video from each original

---

Published in Transactions on Machine Learning Research (10/2024)
Table 1: Recognition results with synthetic audio (A), visual (V), and audio-visual (AV) data on VGGS- 10K. We observe that audio-visual modcling generally outperforms individual oncs, highlighting the necessity of audio-visual data distillation.
Conoect Selertions
Thining Set Synthests
Randonss
Herding
MTT
DM
Whote
IPC
AV
AV
A.
AV
A
AV
A
AV
14.27amom
11.652144
15.44
26.32
14.72as0
20.77am
30.99
21.15a22
34.13
29,00u2ms 20.00a:
38.54,21:
22.7
32.50.am
34.58,13s
28. 98..
30.89
30.372257
25.41a10
36.79
33.00amm
31,03111
43.85
02.07a04 48.192224 68. 68.24aam
20
36, 78a2e
31.06
45.10-2m
Millace
$1.881344
50.20
15.78ates
29.52z1:a
51.87212/
35 38.90
36.28au
49.01:244
Welling
2008
Croeravette et al.
2022
Zhao & Bilen 2023
Table 2: Audio-visual event recognition results for different fusion methods and images per class (TPC) on VGGS-10K. The results are presented over synthetic audio and visual data distilled using audio-only and visual-only DM (Zhao & Bilen 2023). respectively. For fixed IPC, the same synthetic data is used. Ensemble consistently achieves the highest accuracy.
Audio- Visual Pusion
Only-A
Only-V V
Concat
Sum
Attention
Eusemble
29.60
26. 26.10
33.77 6s
24
9.97 2.97
36.5422.2
IPC
33.60
31.6 11.
21.71=127
40.49a1.3
10.1 20.1
43.85
38.93
35.23
46.59_134
46.05
11.10 ss
49.01g2.4
clip of the train/test split and have around 165k/13k samples respectively. For exploratory analyses and experimental setup of this novel task, we randomly selected a subset of 10 classes from VGGSound with 8808 train videos and 444 test videos. This subset. is referred to as VGGS-10K.
MUSIC-21: (Zhao et al., 2019 comprises synchronized audio-visual recordings featuring 21 distinct musical instruments. For our study, we focus exclusively on the solo performances subset and segment cach video clip into discrete, non-overlapping windows of one second. We randomly partition this subset into train/val val /test splits of 10,
samples, respectively.
AVE: Tian et al. 2018) consists of 4,143 video clips spanning over 28 event categories. We segment each clip into non-overlapping one-second windows aligned with the synchronized annotations, resulting in train/ /val/vasttttttttt splits of 27, 17,726/3.288 //0/////28///000 3,305 samples, respectively.
Implementation Details. Following previous distillation methods (Zhao et al. 2023 Zhao & Bilen 2023;
Cazenavette et al. 2002),, we use ConvNet architecture (Zhao et al. 2020) for both audio and visual inputs. The audio ConvNet consists of 3 blocks with convolution, normalization ReLU, and pooling layers. For larger image inputs (224 x 224), use such blocks. We use learning rate of 0.2 and an SGD optimizer with momentum of 0.5. Our synthetic data is initialized with Herding-selected audio-visual (AV) pairs and trained with a batch size of 128. For IPC 1 and 10, we set λςOM and λCAM to 10, and for IPC 20, we set them to 20. Audio is sampled at 11kHz and transformed into 128 x 56 log mel-spectrograms.
Baselines. For DM (Zhao& Bilen 2023), we use the same learning rate, optimizer, and AV pair batch size. For DC (Zhao et al.) (2200), DSA (Zhao & Bilen (021)) and MTT (Cazenavette et al. (022), we extend image-only distillation to independently learn audio and visual distilled data. We randomly select. AV pairs for the random selection method. For Herding, we greedily select samples closest to the cluster center and follow (Wu et al.) 2023) to get AV data by concatenating AV features extracted from models trained on the whole dataset. More details are in Appendix C
Evaluation. We report the mean accuracy and standard deviation of 3 runs where the model randomly initialized and trained for 30 epochs using the learned synthetic data. Each run consists of 5000 iterations and we use a similar setup as (Tian & Xu 2021) to train the audio-visual event recognition models. For reference, we also report the performance of models trained the whole dataset under the same training conditions.
8

---

Published in Transactions on Machine Learning Research (10/2024)
Table 3: Comparison with previous data distillation methods for audio-visual event recognition. Following common practice, we evaluate our method on four different datasets with different numbers of synthetic images per (IP)) Ra)io(%): the ratio of condensed images to the whole training set. Whole Data: the accuracy of the ConvNet model trained on the whole training set and is the upper bound of the performance. ' refers to configurations for which the method couldn't scale up.
Corcset Selection
Training Set Synthesis MTT
IPC
Ratio%
Whole data
Random
Herding!
DC
DSA
DM
Ours
1
0.11
15.44
20.77+2.11
18.28
19.32=1.35
34.1323.02
36.5412..2
40,41 s1
VGGS-10K
10
1.13
32.01=1.61
1..8
32.10.0.8
36.61=1.01
36.79+1.77
13,8521.75
54.99
68.2440.75
20
2.26
45.142.51
50.2+0.74
51.8721.20
49.01g2.41
58.04g1.66
1
0.18
1.38
2.14
1.55
3.08
2.97
VGGSound
10
1.87
5.55g0.19
7.09
6.40.o.2s
8.23
25.5410.19
20
3.74
8.00
9.64
2.64
9.85.0.34
1
0.014
24.12
26. 26.15 2..001
22.60
22.98_1.16
38.26 28.71
44.02,22.21
MUSIC-21
10
0.14
45.77
51.89
42.25
54.78 1.
68.07 1.0.
85.9310.044
20
0.28
54.86
59.981u.85
61 61.06
70.30
1
0.10
10.07=1.16
11.84
10.45
10.7
12.13
21,70
23.00
AVE
10
1.0
20.0$1.45
26, 06.56
23.15
28, 1..10
36.82
52.20
20
2.0
26.
26.32
33.04
32.57=0.97
40.13 1.00
Welling
2009
Zhao et al. 2020
Zhao Bilen.
2021
Cazenavette
2022
Zhao & Bilen 2023
4.2
Experimental Results
Audio-visual integration with distilled data is still helpful. Audio-visual integration has consistently demonstrated superior performance over unimodal data across various tasks for real data. We aim to further investigate whether this advantage extends to audio-visual distilled data for different data distillation approaches. To explore this, we employed audio-visual event recognition as benchmark to compare the performance of unimodal distilled data and audio-visual distilled data. Using the VGGS-10K dataset and an ensemble model trained on individually learned audio-visual data, we evaluate several data distillation methods and different synthetic data sizes. The results, shown in Tab. D clearly demonstrate that audio- visual integration consistently outperforms unimodal modalities in most cases. This observation suggests that effective audio-visual integration remains beneficial even for distilled data.
Multimodal Fusion. Audio-Visusl fusion plays a crucial role in the performance of multimodal models. We investigated whether different audio-visual fusion strategies influence the performance of models trained on distilled synthetic data. To address this question, we compared several standard audio-visual fusion approaches: Sum, Concatenation (Concat), Ensemble, and andio-guided visual attention (Attention) (Pian let al.] 2023). Tab. ? presents the audio-visual recognition accuracy across different synthetic data size settings using separately learned VGGS- 10KK data with DM. From Tab. 2 we can see that the ensemble method consistently outperforms other approaches in all image-per-class scttings, followed by a comparable performance of Concatenation and Sum, and the lowest performance of attention fusion. The comparatively low fusion results in attention fusion can be accounted for classwise alignment losses, spatial distortions (as visualized in Fig. 5) and a larger number of trainable model parameters (More details are in the Appendix). Consequently, we employ the ensemble fusion as the default. Furthermore, these results can also demonstrate that audio-visual integration with synthetic data is stiill helpful when employing different fusion modh
Comparison with Data Distillation Baselines. Tab. 3 compares the performance of our audio-visual distillation method with other baselines across four datasets and three images-per-class (IPC) values. Our audio-visual data distillation approach consistently outperforms vanilla audio-visual distillation with DM, demonstrating the effectiveness of incorporating joint matching losses to strengthen cross-modal alignment. In addition, similar to previous image-only distillation methods (Cazenavelle et al. 2022), we observe di- minishing returns as IPC increases. For instance, in MUSIC-21, there's a significant performance jump from 44.02% to 68.077 08.07% when moving from 1 to 10 IPC, while the improvement from 10 to 20 IPC is more modest, reaching 70.30%. Interestingly, a simple heuristic method like herding becomes competitive starting at IPC 10 and may even outperform DM and MTT in some cases. This could be attributed to the increase in
9

---

Published in Transactions on Machine Learning Research (10/2024)
Table 4: Ablation study IPC=10 for (left) proposed losses and (right) herding and factor.
Random
Herding
Fansor
Base
ICM
CGM
VGGS 10k
AVE.
=1.0
20.00
Random
Herding
Factor
Bose
VGGS- 10k
AVE
11.81
26.8
CGM
40.28=2.84
31.80
43.85
28.14
45.31 45.31=2.06 22.05
34.80
45.87
30.71
45.73_2.m
35.0410sm
44.47
12.36
45. 15.18_m
34.67
46.68
33 13.60
19.07 .1.7
36.13
41.67
31.51
162122
25.78
50.41
15, 35.12_
49.48
35.43
45.31
34.80_111
54.90
36.82_.088
54.99
36.8220.54
MTT
DM
Ours
Figure 6: t-SNE distribution plot of synthetic audio-visual data (IPC=10) learned by DM and Ours(I=1), with same initilization. (green, blue), (red, black) and (purple, yellow) points are the real (audio, visual) points for the first classes of VGGS-10k. The synthetic (audio, visual) data is represented by (* A). We observe that our synthetic audio and visual distributions better resemble the real distributions.
the number of learning parameters due to the high-rcsolution images. Scaling in DM proved more feasible compared to MTT, DC, and DSA. In fact, fitting visual-only distillation using MTT was not possible even with minimal configurations. This due to large memory consumption when unrolling optimization through
SGD steps Cui et al. (2022;; Zhou al. (2022): Cui et al. (2023) for large datasets and higher IPC.
Ablation Study. We developed two novel audio-visual distribution matching losses aimed at more effec- tively distilling essential audio-visual correlations into synthetic data. Additionally, also herding initialization and factor technique. To validate the contributions of these components, conducted an ablation study by systematic addition. The ablation study results on VGGS-10K and AVE are shown in Tab. 4 from which we can see that each of the proposed parts has a positive influence on the final results. These findings uncquivocally demonstrate the effectiveness of our proposed ICM and CGM losses and herding initialization and factor technique in helping the audio-visual data distillation process.
Original
IPC=1
IPC=10
IPC=50
Visualization. To showcase our distilled data, we plot the learned au- dio and visual data in Fig. 5 at different IPCs. We observe that with increased IPC, the synthetic data remains perceptually closer to the origi- nal audio-visual sample. The crtifacts/patchess in the synthetic data arise as the approach distills information from the entire training data into a few synthetic samples. In addition, the spectrogram and image pixels are learnable parameters, making artifacts more prominent settings with smaller IPCs. Despite this, the model achieves high accuracy on unseen real test data, indicating that these artifacts do not negatively impact performance. Furthermore, three classes of VGGS-10k in Fig. G It illustrates how our approach better captures the underlying distribution of the real data.
compare the data distribution of the first
Figure 5: Visualization of dis- tilled data under different IPCs.
Cross-architecture performance. In Tab. we eval- uate how well our distilled data performs on unseen archi- tectures. We observe that method performs DM and MIT all the unseen architectures by large mar- gin, thus highlighting its superior generalization ability. 10
Table 5:
Cross- cross-architecture performance for dis- tilled data on VGGS- 10k and IPC-1.
Evniuation Arcillettuns
CowNNet
VOG11
LoNes
BesNer18
AlexNet
30.0gg:
22,241
DM Ours
36.442222
26.96
26.14.2m
21.89.1
261081188
40.41:151
34.00.1s1
32.37
30.48-1
$1.09.3m

---

Published in Transactions on Machine Learning Research (10/2024)
Table 6: Audio-visual retrieval results on four different datasets. We train audio-visual ConvNet networks with shared classifier and using Random, DM and Our distilled data of IPC 20. Whole data is trained using the entire training data. We observe that our approach helps to distill better audio-visual alignment.
Michod
Reit
pok rest
RCM
MedRi.
Ro1
VGO5Somn
RoM
subset
Medll.
RD011
Mustc test
R:01
Medil.
Rom
ReS0
subset
Medii
Randim
1533
3.30
15.55gs
30.23
12. StaSaa
A-VV
DM Zhao &c Dilen 20220
Ours
19.58
59.55...4
47.3311:77
5.00
6.00
0.90.ass
6.82.035
3.8
4.
11300
110,g.09
51.74,054
16.82.231
s8.01.1.
37. 37.14
2.00
15.09.2.4
6.00
25.00.1.4
32.34.077
11.36
9.00
Whole data
41.0012.0
70/00s300
2,00
1.79u073
70.35
57.14,23
10.s
1,00
27.01,5.35
Rendom
1006,230
0.N
3.00
100.0622
10.10
24 44.12.14
14.0
9.
26 M.
16.00
V-+A
DM
0.09
0.7
4.22,003
7,30
33
10.95
27.33
0.53
0.82.0.3
143.00
143.00.555
27.30 11.0
42.54 1.
1:00.1:5
6,43
34.52.333
10.16.1.99
Winter dats
15335530
700002200
8.00mm0
10.54
10.3
33.80
3.06pa..
17.34
Query
Audio-Visual Visual Retrieval. We have demonstrated that audio-visual distilled data facilitates learning effective audio-visual representa- tions for event recognition. To further examine the audio-visual alignment, we explore whether distilled data could help learn well- coordinated audio-visual space for cross-modal retrieval. Since our audio-visual distillation model focuses on semantic aligument rather than instance-level alignment, we evaluate audio-visual retrieval in a class-wise setting. Following (Gong al. 2022), we create a re- trieval test set by uniformly sampling a subset of five audio-visual samples per class from the original Lest split. We train the audio and visual ConvNet (Surís et al. 2018)) with ArcFace margin loss (Deng) ctt al. 2019). The shared classificr and margin loss help to learn a joint-modal embedding space with angular margins between classes.
Audio-
Tap
Top
Visual
Visual
Visual
Visual Query
Top Audio
Top Audio
3
Audio
Figure 7: Audio-visual retrieval exam- ples. We observe close alignment of audio queries with top visual results and vice versa.
We train the model from scratch using the distilled data of TPC=20 and the same learning setting as the classification model.
We use these trained audio and visual components to get the corresponding representation of test samples and calculate the class retrieval recall at rank 1, 5, and median rank based on the cosine similarity. The results of audio-to-visual and visual-to-audio retrieval, in Tab. G demonstrate that our losses help distill andio-visual alignment (from real data) and hence our method outperforms DM in almost all scenarios. Fig. 7 visualizes the top-3 gross-moda retrieval from the VGG-10k test set. We observe that the retrieved modalities from the same class are closcly aligned when trained with our audio-visual distilled data.
5 Conclusion and Discussion
In this paper, we explore a new task of multimodal distillation using audio-visual data. To evaluate the distilled data, we use audio-visual event rocognition as the proxy task. To tackle this audio-visual dataset distillation problem, we first roduce a vanilla AVDM method that could learn audio- and visual-only distilled data and simply fuse the information for event recognition. We further improve the quality of condensed data by introducing two new losses to strengthen mross-modal alignment. We further show the improved alignment using the classwise cross-modal retrieval task. Experimental results on four audio-visua datasets show that our approach outperforms other methods consistently and audio-visual integration with condensed data is still helpful. This provides a new direction in the dataset distillation domain.
As the first attempt on this new task, our work has some limitations and introduces extents that are beyond the scope of a single paper but represent compelling directions for future research: (1) Compression-accuracy tradeoff:: While our approach significantly reduces training data size, it does so at the cost of a performance drop (w.r.t whole data). Future work could explore methods to improve accuracy, potentially leveraging pre- trained weights (see Appendix F (2) Short video segments: Our exploration focuses short video segments. Extending our approach to long-form vidcos is a promising avenue for future rescarch (see Appendix G (3) Design for other audio-visual lasks: our approach learns synthetic data capturing class-level alignment between audio and visual modalities but lacks the instance-level detail needed for more complex audio- visual tasks. Future rescarch will focus on creating synthetic data with improved instanco-level alignment to enhance training for complex tasks such as audio-visual sound source localization. (4) Large datasets: Although our approach on VGGSound demonstrates better scalability and outperforms the baselines, the
11

---

Published in Transactions on Machine Learning Research (10/2024)
overall performance remains limited. This limitation could be due to the large number of classes and the complex label space that needs to be learned from smaller datasets. This scenario is similar to ImageNet-1K for visual-only dataset distillation Zhou et al. (2022) performed poorly and required additional techniques like soft label aligument and reduced memory Cui et al. (2023) usage to enhance performance. These extensions are outside the scope of our current work; and we aim to tackle them in the future.
Broader Impact Statement
Given that audio-visual datasets may contain personally identifiable information, improper handling, storage, or sharing of distilled datasets could lead to privacy breaches and unauthorized access or misuse of sensitive data. Addressing these privacy concerns through the development of robust safeguards would be crucial.
Acknowledgments
This work was supported in part by an Amazon Research Award Fall 2023. Any opinions, findings, and conclusions or recommendations expressod this matcrial are those of the authors and do not refiect the views of Amazon.
References
Relja Arandjelovic and Andrew Zisserman. Look, listen and learn. In Proceedings of the IEEE international conference on computer vision, pp. 2017.
Relja Arandjelovic and Andrew Zisserman. Objects that sound. In Proceedings of the European conference computer vision (ECCV), pp. 435-451, 2018.
Yusuf Aytar, Carl Vondrick, and Antonio Torralba. Soundnet: Learning sound representations from unla- beled video. Advances in neural information processing systems, 29, 2016.
Francisco Castro, Manuel J Marin-Jiménez, Nicolás Guil, Cordelia Schmid, and Karteek Alahari. End- to-end incremental learning. In Proceedings of the European conference on computer vision (ECCV), pp. 233-248, 2018.
George Cazenavette, Tongzhou Wang, Antonio Torralba, Alexei A Efros, and Jun- Yan Zhu. Dataset distilla- tion by matching training trajectories. In Proceedings of the IEEE/CVE CCF Conference on Computer Vision and Pattern Recognition, pp. 4750-4759, 2022.
Changan Chen, Unnat Jain, Carl Schissler, Sebastia Vicene Amengual Gari, Ziad Al-Halah, Vamsi Krishna Ithapu, Philip Robinson, and Kristen Grauman. Soundspaces: Audio-visual navigation in 3d environ- ments. In Computer Vision -CCCV 2020: 16th Europcan Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part VI 16, pp. 17 36. Springer, 2020a.
Changan Chen, Sagnik Majumder, Ziad Al-Halah, Ruohan Gao, Santhosh Kumar Ramakrishnan, and Kris- ten Grauman. Learning to set waypoints for audio-visual navigation. arXiv preprint arXiv:2008.0983 006622.... 2020b.
Changan Chen, Ziad Al-Halah, and Kristen Grauman. Semantic audio-visual navigation. In Proceedings of the LEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 15516 15525, 2021.
Honglie Chen, Weidi Xie, Andrea Vedaldi, and Andrew Zisserman. Vggsound: A large-scale andio-visual datasct. In ICASSP 2020-2022 IEEE International Conference Acoustics, Speech and Signal Processing (ICASSP), pp. 721 725. IEEE, 2020c.
Justin Cui, Ruochen Wang, Si Si, and Cho-Jui Hsich. De-bench: Dataset condensation benchmark. Advances in Neural Information Processing Systems, 35:810 122 2022.
12

---

Published in Transactions on Machine Learning Research (10/2024)
Justin Cui, Ruochen Wang, Si Si, and Cho-Jui Hsich. Scaling up dataset distillation to imagenet- 1k with constant memory. In International Conference on Machine Learning, pp. 6565-6590. PMLR, 2023.
Jiankang Deng, Jia Guo, Niaman Xue, and Stefanos Zafeiriou. Areface: Additive angular margin loss for deep face recognition. In Proceedings of the IEEE/CVE conference computer vision and pattern recognition, pp. 4690-4699, 2019.
Tian Dong, Bo Zhao, and Lingjuan Lyu. Privacy for free: How does dataset condensation help privacy? In International Conference Machine Learning, pp. 5378-5396. PMLR, 2022.
Chuang Gan, Deng Huang, Hang Zhao, Joshua Tenenbaum, and Antonio Torralba. Music gesture for visual sound separation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10478-10487, 2020.
Ruohan Gao, Rogerio Feris, and Kristen Grauman. Learning to separate object sounds by watching unlabeled video. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 35-53, 2018
Jort F Gemmeke, Daniel PW Ellis, Dylan Freedman, Aren Jansen, Wade Lawrence, R Channing Moore, Manoj Plakal, and Marvin Ritter. Audio set: An ontology and human-labeled dataset for audio events. In 2017 IEEE international conference on acoustics, speech and signal processing (ICASSP), pp. 776 780. IEEE, 2017.
Yuan Gong, Andrew Rouditchenko, Alexander H Liu, David Harwath, Leonid Karlinsky, Hilde Kuehne, and James Glass. Contrastive audio-visual masked autoencoder. arXiv preprint arXiv:2210.07839, 2022.
Arthur Gretton, Karsten M Borgwardt, Malte J Rasch, Bernhard Schölkopf, and Alexander Smola. kernel two-sample lest. The Journal of Machine Learning Research, 13(1):723 773, 2012.
Jianyang Gu, Kai Wang, Wei Jiang, and Yang You. Summarizing stream data for memory-restricted online continual learning. arXiv preprint arXiv:2305.1664 2023.
Ziyao Guo, Kai Wang, George Cazenavette, Hui Li, Kaipeng Zhang, and Yang You. Towards lossless dataset distillation via iificulty-aligned trajectory matching, arXiv preprint arXiv:2310.05773, 2023.
Di Hu, Feiping Nie, and Xuelong Li. Deep multimodal clustering for unsupervised andiovisual learning. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 9248-9257, 2019.
Chun-Yin Huang, Ruinan Jin, Can Zhao, Daguang Xu, and Xiaoxiao Li. Federated virtual learning on heterogeneous data with local-global distillation. arXiv preprint arXiv:2303.02273 2023.
Evangelos Kazakos, Arsha Nagrani, Andrew Zisserman, and Dima Damen. Epic-fusion: Audio-visual tempo- ral binding for egocentric action recognition. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 5492-5501. 2019.
Jang-Hyun Kim, Jinuk Kim, Scong Joon Oh, Sangdoo Yun, Hwanjun Song. Joonhyun Jcong, Jung- Woo Ha, and Hyun Oh Song. Dataset condensation via efficient synthetic-data parameterization. In International Conference Machine Learning, pp. 11102 11118. PMLR, 2022.
Alex Krizhevsky, Geoffrey Hinton, al. Learning multiple layers of features from tiny images. 2009.
Saksham Singh Kushwaha and Magdalena Fuentes. A multimodal prototypical approach for unsupervised sound classification. arXiv preprint arXiv:2306. 12300, 2023.
Ya Le and Xuan Yang. Tiny imagenet visual recognition challenge. CS 231N, 7(7):3, 2015.
Jun-Tae Lee, Mihir Jain, Hyoungwoo Park, and Sungrack Yun. Cross-attentional audio-visual fusion for superissd action localization. In International conference on learning representations, 2020.
Sizhe Li, Yapeng Tian, and Chenliang Xu. Space-time memory network for sounding object localization in 2021.
videos. arXiv preprint
13

---

Published in Transactions on Machine Learning Research (10/2024)
Yan-Bo Lin, Yu-Jhe Li, and Yu-Chiang Frank Wang. Dual-modality seq2seq network for audio-visual event localization. In ICASSP 2019-2019 IEFE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 2002 2006. LEEE, 2019.
Yan-Bo Lin, Yi-Lin Sung, Jie Lei, Mohit Bansal, and Gedas Bertasius. Vision transformers are paramcter- efficient audio-visual learners. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2299-23000 2023.
Yanqing Liu, Jianyang Gu, Kai Wang, Zheng Zhu, Wei Jiang, and Yang You. Dream: Efficient dataset distillation by representative matching. arXiv preprint arXiv:2302.14416, 2023.
Arsha Nagrani, Chen Sun, David Ross, Rahul Sukthankar, Cordelia Schmid, and Andrew Zisserman.
Speech2action: Cross-modal supervision for action recognition. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp- 10317-10326, 2020.
Andrew Owens and Alexei A Efros. Audio-visual scene analysis with self-supervised multisensory features. In Proccodings of the European conference on computer vision (ECCV), pp. 631-648, 2018.
Weiguo Pian, Shentong Mo, Yunhui Guo, and Yapeng Tian. Audio-visual class-incremental learning. In Proceedings of the IEEE/CH /CVF International Conference Computer Vision, pp. 7799-7811 2023.
Noveen Sachdeva and Julian McAuley. Data distillation: A survey. arXiv preprint ar Xiv:2301 .04272, 2023.
Mattia Sangermano, Antonio Carta, Andrea Cossu, and Davide Bacciu. Sample condensation in online continual learning. In 2022 International Joint Conference on Neural Networks (IJCNN), pp. 01-08.
IEEE, 2022.
Didac Surís, Amanda Duarte, Amaia Salvador, Jordi Torres, and Xavier Giró-i Nieto. Cross-modal embed- dings for video and audio retrieval. In Proceedings of the european conference on computer vision (eccv) workshops, pp. 0 0, 2018.
Yapeng Tian and Chenliang Xu. Can audio-visual integration strengthen robustness under multimodal allacks? In Proceedings of the IEEE/CVT conference on computer vision and pattern recognition, pp.
5601-5611, 2021.
Yapeng Tian, Jing Shi, Bochen Li, Zhiyao Duan, and Chenliang Xu. Audio-visual event localization in unconstrained videos. In Proccedings of the European conference on computer vision (ECCV), pp. 247- 263, 2018.
Yapeng Tian, Di Hu, and Chenliang Xu. Cyclic co-learning of sounding object visual grounding and sound separation. In Proccedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2745 2754, 2021.
Margarita Vinaroz and Mi Jung Park. Differentially private kernel inducing points (dp-kip) for privacy- preserving data distillation. arXiv preprint arXiv:2301.13389, 2023.
Kai Wang, Bo Zhao, Xiangyu Peng, Zheng Zhu, Shuo Yang, Shuo Wang, Guan Huang, Hakan Bilen, Xinchao Wang, and Yang You. Cafe: Learning to condense dataset by aligning features. In Proccedings of the ICEE/CVF Conference Computer Vision and Pattern Recognition, pp. 12196-12208 2022.
Tongzhou Wang, Jun-Yar Zhu, Antonio Torralba, and Alexei A Efros. Dataset distillation. arXiv preprint arXiv:1811.10950 2018.
Yake Wei, Di Hu, Yapeng Tian, and Xuelong Li. Learning in audio-visual context: review, analysis, and new perspective. arXiv preprint 200X::2008.09579, 2022.
Max Welling. Herding dynamical weights to learn. In Proceedings of the 26th Annual International Conference Machine Learning, pp. 1121-1128, 2009.
14

---

Published in Transactions on Machine Learning Research (10/2024)
Xindi Wu, Zhiwei Deng, and Olga Russakovsky. Vision-languaae dataset distillation. arXiv:2308.07444 2023.
arXiv preprint
Yu Wu, Linchao Zhu, Yan Yan, and Yi Yang. Dual attention matching for audio-visual event localization. In Proceedings of the IEEE/CVE international conference computer vision, pp. 6292-63020 6300, 2019.
Yuanhao Xiong, Ruochen Wang, Minhao Cheng, Felix Yu, and Cho-Jui Hsieh. Feddm: Iterative distribution matching for communication-effiicein federated learning. In Proceedings the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pp. 16323-16332, 2023.
Jun Xu, Tao Mei, Ting Yao, and Yong Rui. Msr-vtt: A large video description dataset for bridging video and language. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 5288 5296, 2016.
Ruonan Yu, Songhua Liu, and Xinchao Wang. Dataset distillation: A comprehensive review. arXiv preprint arXiv:2301.00114 2023.
Hansong Zhang, Shikun Li, Pengju Wang, Dan Zeng, and Shiming Ge. Echo: Efficient dataset condensation by higher-order distribution alignment. arXiv preprint arXiv:2312.15927, 2023.
Yunhua Zhang, Hazel Doughty, Ling Shao, and Cees GM Snoek. Audio-adaptive activity recognition across video domains. In Proceedings of the IEEE/CW EEE/ Conference on Computer Vision and Pattern Recognition, 2022.
pp.
Bo Zhao and Hakan Bilen. Dataset condensation with differentiable siamese augmentation. In International Conference on Machine Learning, pp. 12674-12685, PMLR, 2021.
Bo Zhao and Hakan Bilen. Dataset condensation with distribution matching. In Proceedings of the IEEE/CVE Winter Conference on Applications of Computer Vision, pp. 6514-6523, 2023.
Bo Zhao, Konda Reddy Mopuri, and Hakan Bilen. Dataset condensation with gradient matching. arXiv preprint arXiv:2006.09929, 2020.
Ganlong Zhao, Guanbin Li, Yipeng Qin, and Yizhou Yu. Improved distribution matching for dataset con- densation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7856-7865, 2023.
Hang Zhao, Chuang Gan, Andrew Rouditchenko, Carl Vondrick, Josh McDermott, and Antonio Torralba. The sound of pixels. Proceedings of the European conference on computer vision (ECCV), pp. 570-586, 2018.
Hang Zhao, Chuang Gan, Wei-Chiu Ma, and Antonio Torralba. The sound of motions. In Proceedings of the IEEE International Conference on Computer Vision, pp. 1735-1744, 2019.
Hang Zhou, Xudong Xu, Dahua Lin, Xiaogang Wang, and Ziwei Liu. Sep-stereo: Visually guided stereophonic audio generation by associating source separation. In Computer Conference, Glasgow, UK, August 23 28, 2020, Proceedings, Part XII 16, pp. 52 69. Springer, 2020.
2020: 16th Europcan
Yongchao Zhou, Ehsan Nezhadarya, and Jimmy Ba. Dataset distillation using neural feature regression. Advances in Neural Information Processing Systems, 35:9813 9827, 2022.
15

---

Published in Transactions on Machine Learning Research (10/2024)
Appendix
In this supplementary, first explain the direct cross-modal distribution losses that fail to align audio-visual synthetic data, our proposed algorithm, more details about herding and factor technique, additional imple- mentation details, fusion methods, audio-visual retrieval, additional experiments and more visualizations.
A Failure Cross-Modal Distribution Matching
In this soction, we delve into two direct cross-modal matching strategics that, contrary to intuition, do en- hance the alignment of audio-visual synthetic data within the audio-visual distribution matching framework. We explored the integration of alignment between audio and visual distilled data by experimenting with two simple mooss-modal losses: Synthetic Cross-Modal Matching (SCMM) and Real-Syuthetic Cross-Modal Matching (RSCMM).
Synthetic Cross-Modal Matching(SCMMM) Given the natural synchronization in real audio-visual data it is logical to hypothesize that implementing a loss function to bring synthetic audio and visual distributions closer would improve the alignment of synthetic audio-visual data. Precisely, we show the SCMM loss in Eq.
Saal
S.
ABD MMMM =
whe (Aw
we (Aw
(11)
S
S
==1
Real-Synthetic Cross-Modal Matching(RSCMMM) Another intuitive way to align synthetic audio- visual data with real audio-visual data to directly match synthetic audio distribution with real visual data and vere-versa. Formally, we show the RSCMM loss in Eq. [4
T
I..
1
CRS =
Vo (Aw
Ve,
(Aw ($)))1
(12)
S
j=1
|Sal
1
C = RS
voo (x²))
vo. (AAw (s$)))
(13)
[Tol
Saa
1=1
j=1
RSSCMM Lue +
Lev
Lay
(14)
We can further combine these losses with the vanilla audio-visual distribution loss (Cow to get a direct cross-modal matching loss, as shown in Eq. 15
Cav combined
Lev Cay SCMM + Cav RSSCMM
(15)
We do ablation study to show the effect of differrnt. loss funetions using one more losses in Tab. We observe negative effect of adding these direct cross-modal losses and observe severe drop in accuracy, This could be explained by the modality gap that is created by using randomly initialized feature extractors (1b, v.)). To visualize the effect of these losses we plot the distribution of the first three classes of VGGS- data (as shown in Fig. g and observe that these losses lead to unstable training with the synthetic data unable to cover the real audio-visual distribution.
16

---

Published in Transactions on Machine Learning Research (10/2024)
Table 7: Ablation study on the different loss terms and different datasets (IPC-10). Here Base, SCMM, and RSCMM are vanilla audio-visual distribution matching, Synthetic Cross-Modal Matching, and Real-
Synthetic Cross-Modal Matching respectively. We observe that adding SCMM and RSCMM to the baseloss results in lower performance. Random Herding Factor
Base
SCMM
RSCMM
VGGS--Ok
AVE
32.01+1.64
20.00+1.15
39.89
26.8 26.8610.52
40.28
31.80 31.80
45.31
34.80 1168
31.1542.87
29.55
V
30.22:3.19
29. 19.19
24.38+2.21
26.13+0.97
Figure 8: Distribution of synthetic audio-visual data (IPC=10 and 1=1) generated by using the direct cross- modal matching approach(i.e. SCMM and RSCMM). (green, (red,black) and (purple,yellow) points are the real (audio, visual) points for the first 3 classes of VGGS-10k. The synthetie (audio, visual) data is represented by (*, Note that the model trained on whole VGGS-10k data is used for extracting the embeddings. We observe significant misalignment of the synthetic audio-visual data, which can be attributed to unstable training losses.
B Herding and Factor technique
Herding iteratively selects data points (features) that are closest the cumulative mean of previously selected points, aiming create a subset that closely represents the overall data distribution. We use the pseudocode in Fig. to get the data subset of IPC size for each class.
mean torch. mean (feaauueee dim=0, keepdim-True)
idx_selected []
idx_left arange (features [haaatrreeeeaaae[]]]] [ toliat() toollatt)))
for i in range (IPC):
det mean- mean) torch, sum (feettrress (ddeereeddddddeeeeeeeeeeeedddd dis = distance (det, features [edx] idx torch. argmin
idx_selected, (pd_ [idx])
del idx_left [idx]
Figure 9: Pseudocode for Herding.
The factor technique (Zhang et al. 2023 Kim et al. 2022 Zhao et al. 2023) aims to increase the number of representations extracted from S without additional storage cost. Specifically, with the factor parameter
17

---

Published in Transactions on Machine Learning Research (10/2024)
Algorithm 1 Audio- Dataset Distillation
Require:
Initial set of syuthetic samples S for C classes, where each class is represented by a subset S, of syuthetic data, deep neural 0g probability distribution over Pag. Ve, parameterized 0, probahility distribution over Pe., differentiable augmentation As parameterized with angmentation parameter distribution Q, training iterations K;
network Ve, parameterized
learning rate 9, loss weights Arear, ACGM.
Input: Roal training
for & == O...., K.K.K do
Sample 0a Pag Su Po. Sample mini-batch pairs B and B and we f2 every class for 0..... C- 1 do
fog
w Ve, (Aw, (x*))
>. Audio real mean embedding
1.00 wE Vo, (d, (Awe (x"))
Visual real mean embedding
S
(r00 wleng de. (x"))
P. Audio synthetic mean embedding
1B21
me w)E Ve, (Awe (x"))
Visual synthetic emhedding
[ng1
Cbuse 1|gg $ 12 1| $:11²
CIOM
1101g + (2))
(S) S0112
ReOM
[1 8.) (f, + $1)1
Compute Luuse AreM frem + AOaM feOM
Update S. S. nV
end for
end for Output: S
being 1, each image s²"h S is factorized into x 7 mini-cxamples and then up-sampled to its original size in training. We show an example for audio datapoint but it can be similarly extended to visual datapoint.
{ { S0 Sa. Saz
txl
Sa:
Up-sample >
Factor
G2
6.
We conduct an ablation study to find the optimal value of in Tab. 8 and observe that = 2 gives the best performance for VGGS-10k at IPC=10. The performance of 1 = 3 is slightly worse as it discards too many details in each partition.
Table 8: Ablation study of the number of partition 1 in our augmentation on VGGS-10k with 10 Img/Chi
Partition (I)
1
x
3 x
Accuracy
45.66 54.99 54.53
To further demonstrate the advantage of our method, we provide the test accuracy with training steps in Fig.
10 We observe that our method out-perform DM at different factor I values and herding- based initialization improves the initial performance.
C Detailed model architecture and hyperparameters
Following previous distillation methods (Zhao et. al. 2023 Zhao & Bilen 2023 Cazenavette et al. 2022 we use the ConvNet architccture model (Zhao al. 2020g for both audio and visual data inputs. The audio CouvNet consists of blocks with each block consisting of 128-kernel convolution layer, instance normalization, ReLU, and average pooling layer. The last average pooling layer is replaced by an adaptive average pooling layer with (7,7) spatial filter to match the dimension of visual embedding (of size 6272) to facilitate joint matching, For the large image input, we use 5 such blocks. We use the same audio and visual model architecture for all the experiments.
For our approach, we use a learning rate of 0.2 for audio and image synthetic data and SGD optimizer with a momentum of 0.5. We initialize our synthetic data with Herding-selected AV pairs(claborater) in the coreset selection section) and use a real audio-visual pair batch size of 128 at each iteration. For images per
18

---

Published in Transactions on Machine Learning Research (10/2024)
HERDING initalization
RANDOM initalization
55
Test Accuracy (%)
50
Test Accuracy (%)
50
45-
40
35
Ours (I=2)
Ours (I=1)
30
200
400
600
800
1000
1200
1400
200
100
000
800
1000
1200
1400
Training Steps
Training Steps
Figure 10: Porformance comparison between Distribution Matching(DM) and our approach with different factor (1) values, across varying training steps for VGGS- 10k dataset with IPC-10. The left and right plots differ in initialization iee herding- herding-based based and random initialization respectively. We observe that herding initialization gives better initialization performance and our approach performs better than DM at different factor values.
while for IPC 20 we keep ArcM=AccMM200. Following the previous works (Zhang et al. 2023 Kim et al. 2022 Zhao et al. 2023) we keep the factor parameter / to 2. We sample audio at 11kHz and transform them into log spettprcss using a hop length of 200 and 128 mel banks, resulting 128x56 size. The whole datasot indicates training on the entire real training set.
(IPCs(IPC) 1, and 10 we keep
During evaluation, the initial learning rate for the audio model is kept 1e-3, the visual part is kept le-4 and for the classifier layers are kept at le-4. The learning rates are lowered by multiplying by 0.1 after every 10 epochs.
C.1 Baseline implementations
Trajectory Matching(MITT), Contrary to DM, MTT involves range of hyperparameters that require optimization. For MTT, retain 20 expert trajectories for each modality independently and conduct hyperparameter search (memory-constrained) across all synthetic images per class (IPC) and dataset configurations. For the Audio-visual MTT, we separately train distilled data for audio-only and visual- only MTT and then combine these datasets using an ensemble fusion to obtain AV results. The best hyperparameters for each configuration are detailed in Tab. 9
DC and DSA. The original methods are proposed for image-only dataset distillation. For audio-visual datatset distillation, extend the method to independently learn synthetic data for both modalities. Further use this distilled data for evaluation. We follow the optimal hyperparameters suggested by the authors (Zhao et al.) 2020: Zhao & Bilen, 2021] and due to large image resolution, it does not scale to visual-only distillation.
D Audio- Visual Fusion
Audio-visual fusion is critical for multimodal model performance. We examined how different fusion strategics affect models trained on distilled synthetic data. We compared several fusion approaches: Sum, Concate- nation (Concat), Ensemble, and audio-guided visual attention (Attention) (Pian et al. 2023).. The fusion function takes in audio feature fa and visual feature for to get fused feature fau, Here are the formulations of the fusion functions:
Sum It directly sums the audio and visual modality features: fov fa f
19

---

Published in Transactions on Machine Learning Research (10/2024)
Table 9: the experiments learning rate of stop size kept 10-5. *_" denotes the configuration for which the method could not scale up.
for best performing Trajectory Matching(MTT) distillation experiments, for all
Syuthetic Steps
Expert epochs
Max Start Epoch
Syuthetic Batch Size
Learning rase
Starting syuthetic
Modlit
IPC
(Pixels)
step size
1
10
2
20
10a
VGGS-10K
10
10
2
2
20
10a
10"
20
2
2
10
10a
10"
VGGSound
10
10
2
20
16
10-3
A-only
20
2
2
2
10
10-3
2
20
10-3
MUSIC-21
2
20
10
10-x
20
2
2
10
10-3
20
20
10-3
AVE
10a
20
10
10*
10
2
5
16
10
VGGS-1OR 10K
10
2
10
10a
10
VGGSound
10
V-only
10a
MUSIC-21
10
2
10a
10-1
20
2
AVE
2
10a
10
20
Concatenation: It directly concatenates the audio and visual modality features: for [f Ensemble: To get the ensemble prediction Pao the predictions from the audio and visual modality features are averaged. Given the audio and visual learnable projection matrices W and W respectively, we define Pau as
pa = Sfftmaalf Wa Dd "altmaall""" pa pe + Dd 2
Attention: Audio-guided visual attention has shown to be an effective mochanism to learn correlations
between audio and visual features adaptively (Pian et al.] 2023: Li et al., 2021 Tian et al. 2018). Given learnable projection matrices Wa Wo, Ua U and @ as the Hadamard product, we can formally define attention fusion fa as:
Score Wo.
Score" tanh(f"W"), w Softmax(Score" Score")
f = @w
rar = tanhi(/UU*++ (fn
D.1 Why does attention fusion perform worst?
We note three reasons for this.
20

---

Published in Transactions on Machine Learning Research (10/2024)
Classwise alignment. Our three losses aim at learning classwise distribution matching (vanilla audio- visual DM loss) and classwise alignment matching (Joint Matching and Modality Gap Matching loss). Our distilled audio-visual data does not have any instance-wise constraints (we observed some performance degra- dation on adding such constraints) and hence training an attention-based fusion which relies on the paired audio-visual correspondence between the training examples, falls short. performance.
Spatial Distortion. Since all the image and audio-spectrogram pixel values are learnable parameters and additional constraint is added to preserve the spatial information from the original initialized data, the distilled audio-visual data loses spatial information like object boundaries and frequency boundaries. Hence while training, the distilled audio gets confused about which part of the distilled image to look at.
Number of parameters. In addition to the classwise alignment and spatial distortion, we observe that the comparatively large mum- ber of model parameters (shown in Tab. [0] and the small condensed data add another hindrance to audio-guided attention perfor- mance.
Table 10: Comparison Learnable Parameters
Fusion
Parameters
Attention 192,798
Concatenation
125,450
Sum
62,730
Ensemble
125,360
E
Audio-visual retrieval
The framework for training Audio-visual retrieval using distilled data shown in Fig. 11 We alternatively train audio and visual encoder with ArcFace loss (Deng al. 2019). The loss can be formulated in eq.
16 Where f is the angle between the audio (or visual) embeddings and mean class embeddings, is the additive angular margin penalty and is the scaling factor. For case, keep 3.0 and m=0.2.
c (0000(Pp, +m)
log
(16)
cos +m)
Σy=15tt e cos 02
Angular Margin Loss
Labels
Visual
Audio Encoder
Encoder
Shared weights
Figure 11:
Framework for Audio-visual retrieval using distilled data. We have shared classifier weights and we use arcface margin loss get discriminative class embeddings.
E.1 Other details
VGGS-10k categories. We create a subset VGGSound et al. 2020ch for initial experiments and analysis. The selected categorics are playing piano, playing acoustic guitar, police radio chatter, toilet flushing, driving buses, chicken crowing, child speech kid speaking, basketball bounce, fireworks banging, and ocean burbling.
21

---

Published in Transactions on Machine Learning Research (10/2024)
Compute resources. We test our experiments on A5000 and A6000 GPUs with 24 GB memory and 48 GB memory respectively.
Ablation study of loss weights XICM, ACGM. We do an abla- tion study over the effect of loss weights ÅreM, AсGM for different IPCs for VGGS-10k dataset and show the results in Tab. [] From the ablation, we observe that the highest accuracy for TPC 1, 10, 20 occur at Aseи ACGM 10, 20, 20 respectively. Subsequently, we choose Arсм = >car= 10 for IPC 1 and λιeм = λCGM = 20 for IPC 20 for all other datasets. However, we observe λIeM = ACGM 10 working better for IPC 10 for other datasets (especially bigger datasets) and hence we keep it 10 for IPC 10. Due to limited com- putational resources, we do not show ablation over other datasets.
Table 11: Ablation study of the loss weights λICM, ACGM for VGGS-10& dataset with different TPC setting. Here XICM ACGM =
10
20
39.07
40.41
39.62
IPC
10
48. 48.92
51.99
56.19
20
49.48
26.99
58.04
F Pretrained initialization
We show that pretraining can help improve the performance of our approach and can further close the gap with the whole-data. Tab. I2 we observe that initializing evaluation model with VGGSound trained weights can reduce the gap with whole-data from 122 3% for AVE data.
Comparison of different initilization for distilled data at IPC=20. Here Pretrained initialization is of model initialized with VGGSound trained models. Random init. Pretrained init.
Table 12:
Whole data
AVE(IPC-20 10
48.7 48.71g131
52.20ga.3
G Extending to 10-sec videos
In Tab. 13 we extend our problem setting to 10 secs videos. We observe that the trends for 10-sec are similar 1-sce problem sctting.
Table 13: Performance comparison for 10 sec video segments
IPC
Random
Herding
DM
Ours
VGGS 10K
25.33+2.01
22.21
33-20.
39.1443.00
20
46 16.10
19.25
49.75-2.4
53.05
AVE
1
9.50
12.81aa0
12.54 nan
17.22 12.22
27. 01
30.00
30.05-2.4
44.98+104
H Additional Visualizations
Figures 12 and visualize the synthetic data for VGGS-10k for IPC=1 with I and respectively. We observe that all the audio-visual distilled data. consists of repeating patterns and is more prominent in the visual data. Due to the repetitive patterns in Fig, 12 (1 1) we ignore the fine-grained details and achieve better performance by utilizing the storage (as in Fig. 13 (I = 2)) without increasing the synthetic size. We also observe similar patterns in Fig. 14 when trained with DM. We further show that the data becomes less far away from the original initialized real data in Figures 15 where we visualize VGGS-10k with
IPC-10. Fig. 16 17 visualizes the audio-visua data learnt through our method for IPC-1 and VGG, Music-21 and AVE datasets, respectively.
22

---

Published in Transactions on Machine Learning Research (10/2024)
Figure 12: Visualization of audio-visual synthetic data for VGGS-10k for IPC=1, learned through proposed Audio Visual Data Distillation approach (1 = 1).
Figure 13: Visualization of audio visual synthetic data for VGGS-10k for IPC=1, learned through proposed Audio-Visual Data Distillation approach (l=2) 2).
Figure 14: Visualization of audio-visual synthetic data for VGGS-10k for IPC-1, learned through DM method.
23

---

Published in Transactions on Machine Learning Research (10/2024)
Figure 15: Visualization of audio-visual synthetic data for VGGS-10k for IPC=10, learned through our proposed Audio- Visual Data Distillation approach
24

---

Published in Transactions on Machine Learning Research (10/2024)
Figure 16: Visualization of audio-visual synthetic data for first 100 classes of VGGSound for TPC=1, learned through our proposed Audio- Visual Data Distillation approach
25

---

Published in Transactions on Machine Learning Research (10/2024)
Figure 17: Visualization of audio-visual synthetic data for Music-21 for IPC=1, learned through our proposed Andio-Visual Data Distillation approach
26

---

Published in Transactions on Machine Learning Research (10/2024)
Figure 18: Visualization of audio visual synthetic data for AVE for IPC=1, learned through our proposod Audio-Visual Data Distillation approach
27


Reviews:
[{"review_id": "5lwq7OKmsS", "reviewer": "Reviewer_JgdC", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. The paper introduces the concept of audio-visual dataset distillation, which is a new and relatively unexplored domain.  \n2. The authors propose various methods such as the cross-modal losses to adapt distribution matching to audio-visual task, which are novel and valuable.  \n3. The experiments span across four major datasets, providing convincing evidence of the method’s effectiveness. The ablation study is also a strength, as it carefully isolates and evaluates the impact of different components in the method (e.g., novel loss functions, herding-based initialization).\n\nWeaknesses:\n\n1. In Table 3, the performance on VGGSound is notably poor. Could this be due to the larger number of classes in VGGSound compared to other datasets? This raises concerns about the scalability of the proposed method on datasets with a high number of classes.\n\n2. Audio-visual classification models typically achieve over 50% accuracy on VGGSound (e.g., [1] reports 65% accuracy). Why does the model trained on the whole dataset in this paper only achieve 25.54% accuracy? This discrepancy requires further clarification.\n\n3. In Related Work, it is unclear why the method proposed by Wu et al. (2023) cannot be extended to audio-visual dataset distillation. Can the authors provide a more detailed explanation? Additionally, can the proposed method for audio-visual distillation be adapted for the image-text domain?\n\n4. According to Fig. 5, when IPC is large, the synthetic images and audio appear quite similar to the original samples. This contrasts with MTT, where distilled images remain highly abstract and may contain information from multiple sources. This raises concerns about whether the proposed method will be effective on large-scale datasets with many concepts that need to be represented in a limited number of samples.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper extends the unimodal dataset distillation into audio-visual dataset distillation, aiming to obtain smaller synthetic datasets while preserving the cross-modal semantic association between audio and visual data. Based on distribution matching (DM) technique, the authors introduce two novel loss functions, implicit cross-matching (ICM) and cross-modal gap matching (CGM), to enhance the alignment between audio and visual modalities. The method is validated through sufficient experiments on four audio-visual datasets, showing significant improvements over unimodal distillation methods.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "No", "audience_explanation": "", "requested_changes": "Critical changes:\n\n1. Please clarify why the performance of VGGSound is weak in the paper.\n\n2. It would be better to discuss the scalability of the method to large-scale datasets in the paper.\n\nStrengthens:\n\n3. Suggested formatting improvements:\n\n    * The sentence starting with \"While dataset distillation techniques...\" is long and complex. Consider splitting it into two sentences or reconstructing it for clarity.\n\n    * The caption of Figure 15 overlaps with the page number \"24\". This should be adjusted.", "broader_impact_concerns": "Concerns are addressed in the Broader Impact Statement section.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "R7YzizTknO", "reviewer": "Reviewer_CF7n", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- The theoretical and empirical analysis supports the construction of the proposed DM-based multimodal DD methods.\n- The well-defined loss functions with demonstrated interpretability and robustness through extensive experiments.\n\nWeaknesses:\n- Some main experiments to verify the effectiveness of the proposed matching losses are missing.\n- See “Requested Changes” for more details.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper mainly progresses and improves the multimodal (audio-visual) dataset distillation task. The author put forward the DM methods tailored for multimodal datasets, which have been summarized into three specific loss functions: Vanilla Audio-Visual Distribution Matching loss, Implicit Cross-Matching loss, and Cross-Modal Gap-Matching loss. These contributions have been verified through two distinct tasks (Audio-visual event recognition and Audio-visual retrieval), demonstrating the robustness and effectiveness of their approach.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Critical:\n- The main experimental results (Table 3) are obtained with the help of the Herding and Factor technique. And there are no ablations regarding these two technologies. Even in Appendix A (Table 7), the best results are obtained with the base loss and the above two technologies. There lack of sufficient experimental evidence on whether the progress of Acc comes from the proposed DM methods or the high-quality initialization technique.\n- The description of “upper bound” used in the contents and tables is inappropriate, as literature[1,2] has proved that the Acc is beyond the whole dataset just using the distilled dataset.\n\nStrengthen:\n- In Algorithm 1, the Update process, the subscript of the updated dataset $\\mathcal{S}_{c}$ should be $c+1$.\n- The failure of MTT on some datasets should be explained more clearly. Is it invalid with the OOM error or not suitable for the task?\n- In Figure 3, the discrepancy gap between the original and distilled pair is different from the visualization results in Figure 5, the author should modify the caption of Figure 3, especially for the last sentence.\n\nMinor:\n- The latex codes within tables are different.\n\n[1] Towards Lossless Dataset Distillation via Difficulty-Aligned Trajectory Matching. ICLR2024.\n\n[2] Prioritize Alignment in Dataset Distillation.", "broader_impact_concerns": "The DD method helps protect the privacy information in the original dataset and the Broader Impact Statement in the submission paper is sufficient enough.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "i4a3XDNkJy", "reviewer": "Reviewer_s2fd", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n1. The motivation is clear, and the problem itself is important for the community. \n\n2. The proposed method is easy to follow.\n\nWeaknesses:\n\n1. The writing of this paper should be improved.\n\n2. Some analytic experiments should be added. See the requested changes below.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This article introduces a new audio-visual dataset distillation task to construct a condensed and representative synthetic audio-visual dataset that maintains the cross-modal semantic association. To achieve this task, the article introduces an audio-visual distribution matching framework and investigates the effectiveness of audio-visual integration and various multi-modal fusion methods on four datasets. The authors provide thorough experimental validation, demonstrating the superiority of their method over existing state-of-the-art distillation methods. The extensive experiments underline the effectiveness of the proposed framework.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. It is suggested to indicate the trainable model modules in Figure 3 or caption. Besides, the initialization of synthetic data is to select a subset of the real data. But why the initialization of synthetic data in Figure 3 is blurred and different from the real images? It should be better to draw the initialized synthetic data separately from the updated synthetic data during the training process, which can better illustrate the initial state and the intermediate updated state of the synthetic data.\n\n2. The naming of the loss is not intuitive enough. \"Joint matching\" and \"modality gap matching\" do not adequately describe the relationships between Rv, Ra, Sv, and Sa. If the names could emphasize “cross modalities” or “cross data spaces”, it would be clearer.\n\n3. The article is somewhat redundant, repeatedly using phrases like \"align the modality gap between real and synthetic data\" and \"enforces cross-modal matching between real and synthetic data.\"\n\n4. Figure 6 demonstrates that the proposed method can better capture the underlying distribution of the real data. However, both the herding and the MMD loss are based on the selection and calculation of the mean center of the real distribution. Is there a rational explanation for why the proposed method can obtain more dispersed distillation data?\n\n5. Some typos: the font for L in Algorithm 1 is incorrect; the Tab index in Appendix E is not displayed.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The reviewers acknowledge that the authors have addressed most concerns, and specific aspects such as the use of Herding and Factor have been highlighted for their significant performance improvement. Additionally, the reviewers appreciate the insights into dataset distillation methods and notes that the revisions, including fixing typos and improving the content, have resolved prior issues.
Decision: Accept (poster)