IW-FCV 2022
Generating Images from Small Datasets
Using Adaptive Point-wise Grouped Convolutions
Mana Takeda1, and Keiji Yanai1
1The University of Electro-Communications, TokyoIntroductionl Deep learning models typically use a large amount of data for training.
l However, the construction of large datasets requires a lot of eﬀort.
l For learning with small datasets, the transfer of prior knowledge using
pretrained models is eﬀective.
l For deep generative models, a method of transferring prior knowledge toanother dataset has also been proposed.
l Noguchi and Harada [12] proposed a new method to generate images from asmall data set by transferring a pre-trained generative model.

1

Objective

Objective
Transfer pre-trained generative models
to achieve image generation from small datasets.

2

Related Work ー Few-shot GANーl In general, GANs require a large number of training samples to produce highquality images.
l Few-shot GANs require a large image dataset such as ImageNet for pre-training
but use a smaller dataset for ﬁne-tuning.

3

Related Work ー Few-shot GANーl Noguchi and Harada[12] Noguchi, A., Harada, T.: Image generation from small datasets via batch statistics adaptation. ICCV. 2019

– It is a method for adapting a pre-trained generative model to datasets from
diﬀerent domains.
– To eﬀectively use the pre-learned knowledge, the weights of theconvolutional layers of the generator are all ﬁxed during ﬁne-tuning.
– Adapt only the scale and shift parameters of the batch normalization (BN)layer to a small dataset.

4Method ー Adaptive Point-wise Grouped Convolution ーl Extending the work of Noguchi and Harada (a), we introduce an Adaptive pointwise grouping convolution for more ﬂexible domain adaptation.

(a) Channel-wise modulation（Noguchi and Harada[10]）(b) Adaptive Point-wiseGrouped Convolution(Ours)(c) Adaptive Point-wiseConvolution

5

Method ー Adaptive Point-wise Grouped Convolution ーl The 1×1 convolutional layer, called point-wise convolution (c), constructs newfeatures by computing linear combinations of input channels.
l However, point-wise convolution has the problem of too many parameters.

(a) Channel-wise modulation（Noguchi and Harada[10]）(b) Adaptive Point-wiseGrouped Convolution(Ours)(c) Adaptive Point-wise
Convolution

6

Method ー Adaptive Point-wise Grouped Convolution ーl The idea of grouping convolution is also applied to Adaptive point-wise
convolution (b) as a way to reduce the number of parameters.
l In grouped convolution, the input feature maps are grouped in the channel
direction, and convolution operations are applied between each group.
→ The number of parameters can be reduced.

(a) Channel-wise modulation（NoguchiとHarada[10]）(b) Adaptive Point-wiseGrouped Convolution(Ours)(c) Adaptive Point-wise
Convolution

7

Method ー Training and Inference ーl The generator is ﬁrst pre-trained on a large dataset such as ImageNet.
l Then, an Adaptive Point-wise Grouped Convolutional layer with corresponding
FC layers is inserted immediately after all the batch normalization layers andﬁne-tuned on a small dataset.
l During inference, a randomly sampled vector 𝑧 based on the standard normal
distribution is fed into the generator to generate a random image.

8

Experiments ー Experiment setupーl Model– BigGAN-128
l Dataset– Human face（ FFHQ Dataset ）– Passion ﬂower（ Oxford 102 ﬂower Dataset ）– African ﬁreﬁnch（ 260 Bird Species Dataset ）– BMW（ Cars Dataset ）l Evaluation metric– KMMD
Evaluation metric when thenumber of test images is small
Human face

Passion flower African ﬁreﬁnch

BMW9Experiment1 ー Comparison with the baseline ーl We compared the quality of the generated images when the number ofgroupings was changed based on Noguchi and Harada.
l The quality of the proposed method improved as the number of parameters
increased.
→Adaptive point-wise convolution increased the variation of feature channels.
Model
Noguchi andHaradaOurs

Parameter
ratio

Number ofdata

12550
100

2.966
2.507
2.509

22550
100

2.944
2.496
2.493

42550
100

2.942
2.491
2.490

82550
100

2.928
2.485
2.487

KMMD(↓)[12]10

Experiment2 ー Experiments with additional datasets ーl We used 25, 50, and 100 images sampled from each of the four datasets andcompared the proposed method with the baseline.
l The proposed method can produce more detailed and higher-quality images
than the baseline.
DatasetPassionflowerAfricanfirefinch

Model

Number ofdataNoguchi andHarada

2550
100

2.976
2.977
2.965

Ours

2550100

2.955
2.960
2.954

Noguchi andHarada2550100

2.965
2.531
2.532

Ours

2550100

2.937
2.493
2.506

KMMD(↓)
[12]

[12]

11

Experiment2 ー Experiments with additional datasets ーl The results of interpolation between two
randomly generated latent vectors areshown.

[12]l The proposed method shows clear, smooth,[12]
and stable completion.

[12]

[12]

12

Conclusionl In this work, we proposed a simple and eﬀective method for generating imagesfrom small datasets.
l By updating only the parameters of Adaptive Point-wise Grouped Conv, a newimage can be generated from a small number of images.
l In the future, the method may be used to generate higher-quality images fromsmaller datasets.

13

14