計測自動制御学会東北支部 第 340 回研究集会 (2022.12.23)
資料番号 340-3

意味推定と深度推定を用いたニューラルネットワークによる
TVCM の分類
Neural Network Models for TV Advertisement Classiﬁcation using
Semantic Segmentation and Depth Estimation◦ 大谷 達哉 ∗ ，森 和好 ∗∗◦Tatsuya OTANI∗ , Kazuyoshi MORI∗∗*会津大学大学院コンピュータ理工学研究科, **会津大学 コンピュータ理工学部*The University of Aizu Graduate School, **The University of Aizu.
キーワード : テレビ CM (TVCM), 畳み込みニューラルネットワーク (CNN), 畳み込みリカレントニューラル
ネットワーク (CRNN), 意味推定 (Semantic Segmentation), 深度推定 (Depth Estimation).

連絡先 : 〒 965-8580 福島県会津若松市一箕町鶴賀字上居合９０ 会津大学コンピュータ理工学部
森 和好， Tel.: (0242)37-2615， Fax.: (0242)37-2747， E-mail: k-mori@u-aizu.ac.jp

1. Introduction

many TVCMs, we propose adding Supplementary

Television commercials (TVCMs) provide various information about products and services to sellthem [1]. TVCMs reﬂect the social situation andtrend at that time. These often aﬀect people andData, like in Figure 1, to support TVCM classiﬁcation. In this paper, we propose to employ Semantic
Segmentation and Depth Estimation as Supplementary Data.

social culture. We consider that investigating many
TVCMs is eﬀective for social analysis. Thus, wehave investigated a semiautomatic system for classifying TVCMs. This implementation has been
based on the models of convolutional neural networks (CNNs) [2] and recurrent neural networks(RNNs) [3]. TVCM classiﬁcation based on image
recognition using convolutional neural networks
essentially needs to prepare many training data, thatis, many TVCMs. Unfortunately, it is diﬃcult tocollect TVCMs because there seems no such kindof open database. Therefore, in order to make theneural network training more eﬃcient with not so–1–Fig. 1One example of Supplementary Data.

2. Neural Network

1st StepData

2nd StepData

3rd Step
Data

RNN
Unit

RNN
Unit

RNN
Unit

The neural network is a computing system modeled on humans on the brain and nervous system,
often used in pattern recognition, such as character recognition and speech recognition. A neuralnetwork consists of multiple node layers, including
an input layer, one or more hidden layers, and anoutput layer.
Fig. 2

2.1

Recurrent Neural Network example.

Convolutional Neural Network

Convolutional Neural Network (CNN) is one ofthe neural networks with deep learning [2,4]. This
is often used for image recognition. CNN consistsof convolutional layers and pooling layers. The
convolutional layer outputs feature maps from theinput array. The pooling layer takes the output of
the convolution layer as input and summarizes theappropriately to data the network has not seen. 1In
addition, the training needs a large amount of data.
However, there is a limit to the number of TVCM
videos we can collect. Therefore, we consider creating data which assist the training of neural networks and input them along with the original. We
call this created data ”Supplementary Data.”
Supplementary Data correspond to the images

values in a relatively small speciﬁc region. In thisresearch, the maximum value is used as a summary.

extracted from the TVCM videos. Figures 1 and 3
show examples of Supplementary Data. We consider it very important how to represent Supple-2.2

Recurrent Neural Network

mentary Data. In the early stages of our research,

Recurrent Neural Network (RNN) is also one

we considered developing a neural network of gen-of the neural networks with deep learning. Thiserators of Supplementary Data, as shown in Figure

method is often used to recognize time-series data.

4. However, U-net [5, 6] and GAN [8], known

Units in the RNN are used recursively. This means

as standard generators, require large amounts of

the output of the units is input into the same unit

training datasets and high-performance GPUs foralong with the next data, as shown in Figure 2.

development. Then, we considered that it was dif-

There are several types of units, and Long Short-

ﬁcult to develop an original Supplementary Dataterm Memory (LSTM) is one of them [3]. LSTMgenerator using our available resources.
Therefore, we considered using the existing struc-

can hold output before the previous turn, but simple

ture to create Supplementary data. Using this dataunits cannot.

aimed to reduce our workload and create a large
amount of good-quality data. In this research, We

3. Supplementary dataconsidered two types of supplementary data.

In classiﬁcation with only images extracted fromTVCMs, the training of a neural network is notgood in accuracy, eﬃciency, and generalization
performance, which means that it could not adapt–2–3.1

How to adapt supplementary data

We need to consider ﬁrst the way to adapt supplementary data to the neural network. Our ﬁrst idea
was to input supplementary data into the neural network as the fourth channel of the original image, as
shown in Figure 5. However, we observed that thiswould not maximize the impact of supplementarydata. We consider that this is because the original images and the supplementary data represent
diﬀerent aspects. Therefore, we decided to input
the Supplementary Data into another independent
CNN and combine its output with the output of theCNN of the original image.
To achieve this, we add the output of the poolinglayer of the CNN of the Supplementary Data, whichFig. 3

Image extracted from a TVCM video.

we call the Supplementary Stream, to the output ofthe pooling layer of the CNN of the original data,
which we call the Original Stream, as shown in
Figure 6. This addition is to increase the valueof the pixels in the original image to which theSupplementary Data points. This method makes
it easier for the neural network model to recognizefeatures.
Supplementary DataClassﬁciation
NetworkVIDEO

OutputResultNeuralNetworkSup data
Generator

Fig. 4 Our ﬁrst idea for classiﬁcation network
structure using the supplementary data generator.

Fig. 5 First idea for adapting supplementary datafor input to the neural network.

–3–3.3

Original Stream

Convolution
Layer

Pooling
Layer

Add

Depth Estimation Supplementary Data

Depth Estimation recognizes the input image andConvolution
Layerestimates the distance from the shooting position toSupplemtary Stream

the object [10]. In this study, we used a model ofConvolution
Layer

Pooling
Layer

monocular depth estimation model, which is called

ConvolutionLayerMidas. Figure 8 is an example of the depth estimaFig. 6data.

CNN structure to adapt supplementarytion. The depth estimation data will enable us toclarify the shapes of objects in front of the image.
We also expect to improve accuracy and training
eﬃciency with this Supplementary Data.

3.2

Semantic Segmentation Supplementary
Data

Semantic segmentation [7] predicts classes ofobjects to which a single pixel belongs shown inFigure 1. We used the DeepLabV3 model withResNet-101 in PyTorch vision API for our semantic segmentation [9]. This model can classify apixel into 21 classes. Figure 7 is an example of thesemantic segmentation.
The Supplementary Data based on the semantic segmentation is to help the neural network toFig. 8

understand the main object of input images. We

An image of depth estimation.

aimed to improve accuracy and training eﬃciency
with this Supplementary Data.

4.

Experiment

In the experiments in this paper, we trained neuralnetwork models on each type of Supplementary
Data. After that, we compared the evaluation ofthese models using a test dataset.

4.1

DatasetWe prepared 600 TVCMs. These have two categories: “Car” and “Drink.” Each category has 300
Fig. 7

An image of semantic segmentation.

TVCMs. We extracted 30 images from one TVCM
video.
For Semantic Segmentation Supplementary Data,
we input the 30 images extracted from one TVCM

–4–

video into the Semantic Segmentation network.

4.3

The raw output of the Semantic Segmentation network is the label numbers of the classes. If thisoutput were directly used, the diﬀerence in values
would represent some speciﬁc meaning. Therefore,
we decided to use a color palette to represent theclasses.

Training

We used 600 TVCMs we collected and divided
them into ﬁve groups. Four groups were used fortraining, and the remaining one group was used as
the evaluation dataset. We trained and evaluatedﬁve diﬀerent dataset combinations for training andtesting. We then compute the average of the test

For Depth Estimation Supplementary Data, weinput the 30 images extracted from one TVCM
video into the Depth Estimation network. The out-results for each type of data set. We forced theneural network to train for 25 epochs. The batch
size is 32.

put values are normalized between 0 and 1.

4.2

5. Result and Conclusion

Neural Network Model

Models using Semantic Segmentation Supple-In our approach, we used Convolutional Recurrent Neural Network (CRNN) as in Figure 9. Images extracted from an input video are processedby convolution and pooling layers as the ﬁrst step.
Feature maps are passed to the input of LSTM units.
After that, full-connected layers output the classiﬁcation result. The supplementary data is input intoanother stream separated from the original image
stream. Output values of each pooling layer on theSupplementary Stream are added to the output ofthe pooling layer on the original stream of the samesize. About LSTM, we used the Backward LSTM
and Bi-LSTM [3]. We consider that this structure can ﬁnd highly accurate temporal features because humans sometimes understand the meaning

mentary Data and Depth Estimation Supplementary Data were trained successfully, as shown in
Figures 10, 11, and 12. In our experiment, results
shown in Table 1 show that the model using Semantic Segmentation Supplementary Data produced
better accuracy results than the model using Depth
Estimation Supplementary Data. We consideredthat this result was caused by the fact that the outline of the objects represented by the Depth Estimation Neural Network is often unclear. Models using
Supplementary Data produced more accurate classiﬁcations than the model without Supplementary
Data. According to eﬃciency, the results in Table 2
show that the number of parameters of each modelusing Supplementary Data increased by 12%, andof commercials by going backward.

the training time increased by 50% than models
Original Images

Supplementary Information

Supplementary Data

without Supplementary Data. However, about theConvolution
andPooling
Recurrenct
Unit

Claasiﬁication
Network

number of epochs required for the accuracy rate
to exceed 90% for the ﬁrst time during training,Convolution
andPoolingas shown in Table 3, the models using Semantic
Segmentation Supplementary Data and Depth Es-Fig. 9 Neural network structure with supplementary data.

timation Segmentation Supplementary Data wereless than the model without Supplementary data.
Therefore, we considered that supplementary data–5–

is eﬀective in terms of learning eﬃciency and accuracy in our neural network structure.

9) DeeplabV3resnet101,
https://pytorch.org/hub/pytorch vision deeplabv3 resnet101/10) R. Ranftl, K. Lasinger, D. Hafner, K. Schindler,
V. Koltun, Towards Robust Monocular Depth Estimation: Mixing Datasets for Zero-shot Crossdataset Transfer, 2020, arXiv:1907.01341v3 .

6. Future WorkIn this research, we found that Supplementary
Data is eﬀective in classifying TVCMs. Thus, itis necessary to consider the structure of the neuralnetworks that can use Supplementary Data more
eﬀectively. In particular, we need to improve theCNN part by referring to structures such as VGG[11] and ResNet [12]. In addition, we classiﬁed
two classes in this research: ”Car” and ”Drink.” Itis necessary to add other classes and evaluate them,such as Food, Cosmetic items, and more.

References
1) 山田 奨治, テレビ・ コマーシャルと文化研究 (Television commercial as a resource for Japanese studies: past and present), 日本研究, 35(5), pp.527–536, 2007.
2) A. Krizhevsky, I. Sutskever, G.E. Hinton, ImageNet classiﬁcation with deep convolutional neural networks, 2017.
3) I.J. Goodfellow, Y. Bengio, A. Courville, Deep
Learning, MIT Press 2016.
4) Y. LeCun, Y. Bengio, G. Hinton, Deep Learning, Nature, 521(5), pp.436–444, 2015, DOI:
10.1038/nature14539.
5) O. Ronneberger, P. Fischer, T. Brox, U-Net: Convolutional Networks for Biomedical Image Segmentation, 2015, arXiv:1505.04597 .
6) Z. Zhang, Q. Liu, Y. Wang, Road Extraction by
Deep Residual U-Net, IEEE Geoscience and Remote Sensing Letters, 15(5), pp.749–753, 2018,
DOI: 10.1109/LGRS.2018.2802944.
7) L.C. Chen., Y. Zhu, G. Papandreou, F. Schroﬀ,
H. Adam, Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentation, Proceedings of the European conference on
computer vision 2018 (ECCV 2018, LNCS 11211),
2018, DOI: 10.1007/978-3-030-01234-2 49.
8) I.J. Goodfellow, J. Pouget-Abadie, M. Mirza,
B. Xu, D. Warde-Farley, S. Ozair, A. Courville,
Y. Bengio, Generative Adversarial Networks,
2014, arXiv:1406.2661 .

–6–11) K. Simonyan, A. Zisserman, Very Deep Convolutional Networks for Large-Scale Image Recognition, 2015, arXiv:1409.1556 .
12) K. He, X. Zhang, S. Ren, J. Sun, Deep
Residual Learning for Image Recognition, 2015,arXiv:1512.03385 .

Fig. 10Training progress of model without Supplementary Data.

Architecture

Loss(average)

Accuracy(average)

No Supplementary Data
Semantic Segmentation Supplementary Data
Depth Estimation Supplementary Data

0.58
0.42
0.70

0.83
0.93
0.88

Table 1

Averages of the evaluation score.

Architecture

Parameters

Training Time(seconds)

No Supplementary Data
Semantic Segmentation Supplementary Data
Depth Estimation Supplementary Data

23,871,810
26,734,082
26,733,506

128
177
180

Table 2Trainable parameters and training time.

–7–Fig. 11

Training progress of model using Semantic Segmentation Supplementary Data.

Fig. 12

Table 3Training progress of model using Depth Estimation Supplementary Data.

Architecture

Step over 90% accuracy

No Supplementary Data
Semantic Segmentation Supplementary Data
Depth Estimation Supplementary Data

13.60
3.74
8.28

The average number of epochs required for Accuracy rate to exceed 90% during training.

–8–