Dataset Distillation for Medical Dataset Sharing
Guang Li, Ren Togo, Takahiro Ogawa, Miki Haseyama
Graduate School of Information Science and Technology, Hokkaido University

AAAI-23
Paper ID: 10INTRODUCTION
Medical data sharing

Problems➢ Although a large medical facility may
provide adequate healthcare information, asingle small medical facility does not havethe capacity.

➢ Problem 1: Privacy protection has been
a severe issue hindering the process
when sharing medical image datasetsfrom different hospitals.

➢ It is necessary to obtain adequate
healthcare information through transmission
between multiple hospitals.

➢ Problem 2: Sharing large-scale highresolution medical image datasets
increases transmission and storage
costs.

➢ The sharing of medical data plays anessential role in enabling the cross-agency
flow of healthcare information and improves
the quality of medical services [1].

➢ The solution to these problems will
significantly promote the development of
medical dataset sharing [2].

A method to solve existing problems is needed.

Medical dataset sharing between hospitals is needed.

PROPOSED METHOD
Concept

The objective of our method is to have the parameters of thestudent network trained on the distilled dataset match theparameters of the teacher networks trained on the original
dataset.
Before the distillation process, we first train many teacher
networks on the original COVID-19 dataset 𝒟 and obtain theirparameters.
Then we perform gradient descent updates on the student
parameters with respect to the cross-entropy loss of the distilled
dataset 𝒟c .

MethodIf the similarity of parameters in 𝜃෨𝑖+𝐽 and 𝜃𝑖+𝐾 is less than athreshold 𝜎, these parameters are recognized as difficult-tomatch parameters and will be pruned.
The final loss ℒ calculate the normalized L2 error between the෩ 𝑖+𝐽 and teacher parameters 𝜃′𝑖+𝐾 .
pruned student parameters 𝜃′Finally, we minimize the loss and backpropagate the gradientthrough all updates to the student network for obtaining the
optimized distilled dataset 𝒟c .
Novelty: We improve the distillation performance by introducing
parameter pruning; With the proposed method, the sharing of
medical datasets between different hospitals will be more
efficient and secure.

𝒟Our method can achieve high COVID-19 detection accuracy even when using scarce distilled chest X-ray images.

EXPERIMENTAL RESULTS
DatasetQuantitative evaluation
Ours

The largest open COVID-19 CXR dataset [3]
Train
2,893
4,810
8,154
1,076

Test
723
1,202
2,038
269

BYOL

SimSiam

MAE

Transfer

84.1%

85%

Ground Truth
C: COVID-19
L: Lung Opacity
N: Normal
V: Viral Pneumonia

SKD

90%

82.7%

80%

Accuracy

Class Total
C
3,616
L
6,012
N
10,192
V
1,345

MTT

74.2%

75%
70%

68.3%
66.8%

65%

62.3%

60%

Settings

53.9%

55%
50%

Dataset distillation:➢ The number of teacher networks: 100➢ Threshold 𝜎 : 0.1➢ The number of distilled data: 20 images per class➢ Network structure: 128-width 5-depth ConvNet➢ Evaluation: 4-class accuracy

Qualitative evaluation
Real image
For example, soft distillation techniques have already impacted medical
AI research, which trains its models using sensitive health information. In
one recent paper [1], researchers used soft distillation in diagnostic xray imagery based on a small, privacy-preserving data set.

Comparative methods:➢ MTT: Matching training trajectories➢ SKD: Self-knowledge distillation based selfsupervised learning➢ BYOL: Bootstrap your own latent➢ SimSiam: Simple Siamese representation learning➢ MAE: Masked Autoencoder➢ Transfer learning from ImageNet

Distilled imageCLNV➢ Covered by Scientific American, Deccan Herald,
日本経済新聞, 环球科学, 机器之心, 腾讯新闻, 网易新闻➢ Cited at TPAMI, NeurIPS, ICML, ICLR, CVPR
➢ Awesome-Dataset-Distillation (GitHub: 540 stars)➢ Most Popular AI Research Aug 2022

[1] Guang Li, et al., “Soft-label anonymous gastric X-ray image distillation,” in IEEE ICIP, pp. 305-309, 2020.
[2] Guang Li, et al., “Compressed gastric image generation based on soft-label dataset distillation for medical data sharing,” Elsevier CMPB, vol. 227, 107189, 2022.
[3] Guang Li, et al., “Self-knowledge distillation based self-supervised learning for COVID-19 detection from chest X-ray images,” in IEEE ICASSP, pp. 1371-1375, 2022.
This study was partly supported by AMED Grant Number JP21zf0127004, the Hokkaido University-Hitachi Collaborative Education and Research Support Program, and the MEXT D-Drive-HU Program.