ҟͳΔղ૾౓ؒͷ੔߹ੑΛߟྀͨ͠‫ݦ‬ஶੑ༧ଌϞσϧʹؔ͢Δ‫ڀݚ‬ࢦಋ‫ڭ‬तɿࢁԼོٛ

TP19010 ੉ඌढ़‫و‬
1. ͸͡Ίʹ‫ݦ‬ஶੑ༧ଌ͸ɼਓ͕‫ڵ‬ຯɾؔ৺Λ࣋ͭͱߟ͑ΒΕΔྖҬΛώʔτϚοϓʹΑΓද‫͢ݱ‬Δɽ‫ݦ‬ஶੑ༧ଌͷΞϓϩʔνͱͯ͠ɼपғͱ͸ҟͳΔ৭࠼΍Τοδಛ௃Λநग़ͯ͠‫ݦ‬ஶྖҬΛਪఆ͢Δख๏͕͋Δɽ·ͨɼConvolutional Neural
Network (CNN) ͷൃలʹΑΓɼਓͷࢹઢ͔ΒಘΒΕͨ‫ݦ‬ஶੑϚοϓΛֶशɾਪ࿦ʹ༻͍Δߴਫ਼౓ͳ‫ݦ‬ஶੑ༧ଌख๏΋ఏҊ͞Ε͍ͯΔɽҰํͰɼ‫ݦ‬ஶੑ༧ଌͷԠ༻ઌͱͯࣗ͠ಈӡసγεςϜΛ૝ఆ͢Δͱɼ༧ଌਫ਼౓͚ͩͰ͸ͳ͘ϝϞϦফඅྔ΍‫ؒ࣌ࢉܭ‬ͷ୹ॖ͕ॏཁͱͳΔɽͦ͜Ͱɼຊ‫ڀݚ‬Ͱ͸ϝϞϦফඅྔΛޮ཰Խͨ͠‫ݦ‬ஶੑ༧ଌϞσϧΛఏҊ͢
Δɽ͞Βʹɼύϥϝʔλͷগͳ͍ϞσϧͰߴਫ਼౓ʹਪఆ͢ΔͨΊʹɼը૾ղ૾౓ຖͷ‫ݦ‬ஶੑͷҰ؏ੑΛߟֶྀͨ͠शख๏΋ఏҊ͢Δɽ

2. ‫ݦ‬ஶੑ༧ଌ‫ݦ‬ஶੑ༧ଌͷ୅දతͳख๏ͱͯ͠ɼRARE2012 [1] ͕‫ڍ‬͛ΒΕΔɽRARE2012 Ͱ͸ɼओ੒෼෼ੳʹΑΓΧϥʔνϟϯωϧຖʹന৭ԽΛߦ͏ɽͦͷ‫ޙ‬ɼಘΒΕͨը૾ʹରͯ͠ϚϧνεέʔϧͷΨϘʔϧϑΟϧλͰಛ௃ྔΛநग़͢Δɽ࠷‫ʹޙ‬ɼಘΒΕͨಛ௃ྔͷ֬཰ີ౓Λ‫͠ࢉܭ‬ɼࣗ‫ݾ‬৘ใྔʹ‫ݦ͍ͯͮج‬ஶੑ༧ଌΛߦ͏ɽͨͩ͠ɼRARE2012 Ͱ͸‫ة‬‫ݥ‬༧஌ͳͲਓͷࣄલ஌ࣝʹ΋ͱͮ͘‫ݦ‬ஶੑͷҧ͍ΛߟྀͰ͖ͳ͍ͱ͍͏໰୊͕͋Δɽ·ͨɼਓͷࢹઢ৘ใΛར༻ͨ͠CNN ʹΑΔߴਫ਼౓ͳ‫ݦ‬ஶੑਪఆख๏͕͋ΔɽSalNet [2] Ͱ͸ɼ৞ΈࠐΈ૚ 3 ૚ɼશ݁߹૚ 2 ૚͔Βߏ੒͞ΕΔωοτϫʔΫΛ༻͍ͯ‫ݦ‬ஶੑϚοϓΛग़ྗ͢ΔɽωοτϫʔΫͷֶशʹ͸ਓͷࢹઢσʔλΛ΋ͱʹͨ͠ϚοϓΛ‫ͯ͠ͱࢣڭ‬༻͍Δɼ·ͨɼEML-Net [3] Ͱ͸ɼҟͳΔσʔληοτͰࣄલֶशͨ͠ෳ਺ωοτϫʔΫͷࢹ֮ಛ௃Λ౷߹͢Δ͜ͱ
Ͱߴਫ਼౓ͳ༧ଌ͕ՄೳͰ͋Δɽ͔͠͠ɼEML-Net ͸ɼେ‫ن‬໛ͳωοτϫʔΫΛར༻ͯ͠ಛ௃நग़Λߦ͏ͨΊɼϞσϧͷύϥϝʔλ਺͕๲େͱͳΓɼ‫͔͔͕ؒ࣌ࢉܭ‬Δ໰୊͕͋Δɽ

3. ఏҊख๏ຊ‫Ͱڀݚ‬͸ɼϝϞϦফඅྔ‫ؒ࣌ࢉܭͼٴ‬ͷ୹ॖͱ༧ଌਫ਼౓ͷ޲্Λཱ྆͢Δ‫ݦ‬ஶੑ༧ଌख๏ΛఏҊ͢Δɽ·ͣɼϝϞϦফඅྔͱ‫ؒ࣌ࢉܭ‬ͷ୹ॖͷͨΊʹɼMobileNetV2 ͱMixed Depthwise Convolution Λར༻ͯ͠ωοτϫʔΫΛߏங͢Δɽ͜ΕʹΑΓɼύϥϝʔλ਺Λ࡟‫͖Ͱݮ‬Δɽ࣍ʹɼ༧ଌਫ਼౓Λ޲্ͤ͞ΔͨΊʹɼਓͷࢹ֮తಛੑʹ‫ͮج‬͍ͨղ૾౓ຖͷࢹ֮తಛ௃Λޮ཰Αֶ͘श͢Δଛࣦؔ਺ΛఏҊ͢Δɽը૾ղ૾౓ͱ‫ݦ‬ஶੑϚοϓͷؔ܎ੑʹ͍ͭͯɼਓͷࢹઢΛऩू͠ௐࠪͨ͠‫͕͋ڀݚ‬Δ [4]ɽ͜ΕʹΑΔͱɼղ૾౓Λ32 ෼ͷ 1 ·ͰϦαΠζͨ͠ը૾ͷ‫ݦ‬ஶੑϚοϓ͸‫ݩ‬ͷղ૾౓ͷ‫ݦ‬ஶੑϚοϓͱඇৗʹ૬͕ؔߴ͍͜ͱ͕֬ೝ͞Ε͍ͯΔɽ͜ͷ஌‫ݟ‬Λ‫͔͠׆‬ɼ‫ݩ‬ͷղ૾౓͔Β༧ଌ͞ΕΔ‫ݦ‬ஶੑϚοϓͱ௿ղ૾౓Խͨ͠ը૾͔Β༧ଌ͞ΕΔ‫ݦ‬ஶੑϚοϓͷ੔߹ੑΛଛࣦͱͯ͠‫ٻ‬ΊΔɽ͜ΕΛఆࣜԽ͢Δͱࣜ (1)ͷΑ͏ʹఆٛͰ͖Δɽ

Lres = BCE(S GT , S Pb ) +N!

BCE(S Pb , S P2i )(1)i=1ୈҰ߲͸ɼόΠφϦΫϩεΤϯτϩϐʔΛ༻͍ͨਅ஋ͱ‫ݩ‬ͷղ૾౓͔Β༧ଌ͞ΕΔ‫ݦ‬ஶੑϚοϓͷଛࣦͰ͋Δɽୈೋ߲͸ɼ‫ݩ‬ͷղ૾౓͔Β༧ଌ͞ΕΔ‫ݦ‬ஶੑϚοϓͱ௿ղ૾౓Խͨ͠ը૾͔Β༧ଌ͞ΕΔ‫ݦ‬ஶੑϚοϓ͔ΒͷଛࣦͰ͋Δɽ͜͜ͰɼBCE ͸όΠφϦΫϩεΤϯτϩϐʔɼS GT͸‫ݦ‬ஶੑϚοϓͷਅ஋ɼS Pb ͸‫ݩ‬ͷղ૾౓͔Β༧ଌ͞Εͨ‫ݦ‬ஶੑϚοϓɼS P2i ͸ղ૾౓Λ 2i ෼ͷ 1 ʹͨ࣌͠ͷਪ࿦
݁ՌɼN ͸μ΢ϯαϯϓϧ͢Δճ਺Ͱ͋Δɽਤ 2 ʹࣔ͢Α͏ʹɼ௿ղ૾౓ͷը૾͸‫ݩ‬ͷղ૾౓ͷը૾Λ 2i ෼ͷ 1 ʹμ΢ϯαϯϓϦϯά‫ޙ‬ɼόΠϦχΞิؒʹΑΓ‫ݩ‬ͷղ૾౓΁ΞοϓαϯϓϦϯάͨ͠ը૾ͱ͠ɼϞσϧʹೖྗ͢Δɽ
32646432128

IRL
BN

132

ReLU6×364
128

256

256

128!"#(% &01 ,% &' )

512

IRL
BNReLU6×4High resolution
1/2-% &'% ./!"#(% &' , % &)* )Weight share32646432

128

IRLఏҊख๏Ͱ༻͍ΔωοτϫʔΫߏ଄Λਤ 1 ʹࣔ͢ɽ͸͡ΊʹɼMobileNetV2 Λಛ௃நग़‫ͯ͠ͱث‬ը૾͔Βಛ௃ΛಘΔɽ͜ͷ࣌ɼMobileNetV2 ͸ IRLɼBatch Normalization(BN)ɼReLU6 Λ૊Έ߹Θͤͨ‫ ܭ‬7 ૚ͷߏ੒Ͱ͋Γɼલ൒ͷ 3 ૚ͱ‫ޙ‬൒ͷ 4 ૚͔Βಛ௃ΛಘΔɽ

BN132

ReLU6×364128256256128512

IRLBNReLU6Low resolution×4&% )*ਤ 2 : ֤ղ૾౓ؒͷ੔߹ੑΛߟྀͨ͠Ұ؏ੑଛࣦ

MixConv

Input
3 332645 564

4. ධՁ࣮‫ݧ‬

7 7ఏҊख๏ͷ༗ޮੑΛࣔͨ͢Ίʹɼ੩ࢭըʹ͓͚Δਓͷࢹ
ઢσʔλ͔ΒಘΒΕͨ‫ݦ‬ஶੑΛऩूͨ͠σʔληοτΛ༻͍ͯɼධՁ࣮‫ݧ‬Λߦ͏ɽ

32128

IRL
BN132

ReLU6×3256256512

IRL

4.1 ࣮‫֓ݧ‬ཁ

160128OutputBNGTReLU6High feature

3.2 ଛࣦؔ਺

3.1 ωοτϫʔΫߏ଄

Low featureғͱ͸ҟͳΔ৭৘ใ΍ࣝผʹ༗༻ͳΤοδͳͲͷ୯७ͳࢹ֮৘ใͱͳΔɽ·ͨɼ‫ޙ‬൒ͷಛ௃͸ɼਂ͍૚͔ΒಘΒΕΔಛ༗ͷಛ௃Ͱ͋Δ෺ମͳͲͷେҬతͳࢹ֮৘ใͱͳΔɽ࣍ʹɼಘΒΕͨಛ௃Λ Refinement layer ʹೖྗ͢ΔɽRefinementlayer ͸ಘΒΕͨಛ௃Λ໌ྎԽ͢ΔϞδϡʔϧͱͳ͓ͬͯΓɼೖग़ྗͷղ૾౓Λมߋ͠ͳ͍ Encoder-Decoder ߏ଄͔Βߏ੒͞Ε͍ͯΔɽ͞ΒʹɼRefinement layer ʹ͸༷ʑͳड༰໺Λ 1 ౓ͷ৞ΈࠐΈ૚ʹू໿Λߦͬͨ Mixed-depthwise
Convolution (MixConv) Λ༻͍Δ͜ͱͰɼԋࢉྔΛ཈͑ͭͭ΋ϩόετͳಛ௃ͷநग़Λߦ͏ɽ࠷‫ʹޙ‬ɼRefinement
layer ͔ΒಘΒΕͨ̎ͭͷಛ௃Λ౷߹͠ɼDeconvolution ૚Λ༻͍ͯ‫ݦ‬ஶੑϚοϓͷਪ࿦Λߦ͏ɽ

×4
Feature Extractor(MobileNetv2)

Refinement layer

Decode layerਤ 1 : ఏҊख๏ͷωοτϫʔΫߏ଄લ൒ͷಛ௃͸ɼઙ͍૚͔ΒಘΒΕΔಛ༗ͷಛ௃Ͱ͋ΔपධՁσʔλʹ͸ɼSALICON ‫ ͼٴ‬CAT2000 Λ༻͍Δɽ
SALICON ͸ɼ༷ʑͳࣗવը૾͔Βߏ੒͞ΕΔ MS-COCOσʔληοτ͔Β 20,000 ຕΛൈਮ͠ɼ໿ 60 ໊ͷඃ‫͔ऀݧ‬Β‫ݦ‬ஶੑΛ֫ಘͯ͠෇༩ͨ͠΋ͷͰ͋Δɽ·ͨɼCAT2000͸ 120 ໊ͷඃ‫ऀݧ‬Λର৅ʹɼອըɼΞʔτɼΦϒδΣΫτɼ௿ղ૾౓ը૾ɼ԰಺ɼ԰֎ɼϥϯμϜͳը૾ɼઢըͱ͍ͬͨҟͳΔλΠϓͷγʔϯΛΧόʔ͠ɼ20 ͷΧςΰϦ͔Βߏ੒͞Ε͍ͯΔɽը૾ͷೖྗαΠζ͸ 640 ʷ 480 Ͱද 1 : SALICONɼ‫ ͼٴ‬CAT2000 σʔληοτʹΑΔఆྔతධՁ݁Ռ

EML-Net[1]Vanilla
Vanilla+R
Vanilla+R+Lres

Params47.08M
4.72M
5.75M
5.75M

SIM ˢ
0.765
0.686
0.714
0.742

SALICON
CC ˢ AUC ˢ
0.878
0.864
0.779
0.844
0.811
0.850
0.847
0.861

NSS ˢ
1.987
1.577
1.673
1.762

KL ˣ
0.520
0.393
0.338
0.282

Params47.08M
4.72M
5.75M
5.75M

SIM ˢ
0.782
0.740
0.757
0.776

CAT2000
CC ˢ AUC ˢ
0.885
0.866
0.833
0.858
0.842
0.860
0.851
0.866

NSS ˢ
2.060
1.731
1.799
1.875

KL ˣ0.298
0.321
0.311
0.299

HighLow
GTVanillaVanilla+RVanilla+R+!"#$

EML-Netਤ 3 : ֤ख๏ʹΑΔ‫ݦ‬ஶੑϚοϓͷਪఆྫ
͋ΔɽSAICON ͸ֶश༻ʹ 10,000 ຕɼධՁ༻ʹ 5,000 ຕΛɼCAT2000 ͸ֶश༻ʹ 1,600 ຕɼධՁ༻ʹ 400 ຕΛ༻͍Δɽ·ͨɼֶशϞσϧͷ൚Խੑೳͷ‫͔఺؍‬Βલॲཧͱͯ͠ೖྗը૾Λ [0,1] ΁ਖ਼‫ن‬Խɼ‫ࠨͼٴ‬ӈ൓సͷσʔλ֦ுΛߦ͍ͬͯΔɽධՁʹར༻͢Δैདྷख๏ͱϞσϧΛҎԼͷΑ͏ʹఆٛ͢Δɽ
EML − Net : EML-Net ͸ɼෳ਺ͷࣄલֶशࡁΈϞσϧΛ‫ֶͨ͠༻׆‬शΛߦ͏͜ͱͰɼࣄલʹΤϯίʔυ͞Εͨࣝผʹ༗ޮͳಛ௃ΛޮՌతʹར༻͢Δख๏Ͱ͋Δɽ
Vanilla : MobileNetV2 ͷσίʔυ෦Λશ݁߹૚͔Β Deconvoliution ૚Λ 3 ૚ʹมߋ͢Δ͜ͱͰɼ‫ݦ‬ஶੑ༧ଌ༻ʹωοτϫʔΫΛมߋͨ͠ϞσϧͰ͋Δɽ
Vanilla + R(Ours) : Refinement layer ‫ ͼٴ‬MixConv Λ‫֫ͯ͠༻׆‬ಘͨ͠ಛ௃Λ౷߹ͯ͠ Deconvolution ૚΁ೖྗ͢ΔϞσϧͰ͋Δɽ
Vanilla + R + Lres (Ours) : Vanilla+R ͷϞσϧʹରͯ͠ɼ֤ղ૾౓ͷ੔߹ੑΛߟྀͨ͠ଛࣦؔ਺ Lres Λಋೖͨ͠ϞσϧͰ͋Δɽ

4.2 ϕʔεϥΠϯͱͷఆྔతɾఆੑతͳൺֱ‫ݦ‬ஶੑ༧ଌʹ͓͚ΔఆྔతධՁࢦඪͱͯ͠ɼSIM(Similarity)ɼCC(Correlation coeﬃcient)ɼAUC(Area UnderCurve)ɼNSS(Normalized Scanpath Saliency)ɼKL(KLdivergence) ͕͋Δɽ͜ΕΒͷධՁࢦඪΛ༻͍ͯɼSALICON σʔληοτͱ CAT2000 σʔληοτʹ͓͚Δ֤ਫ਼౓ͷൺֱ݁Ռͱύϥϝʔλ਺Λද 1 ʹࣔ͢ɽఏҊख๏͸ɼ੔߹ੑΛอͭଛࣦؔ਺ Lres ͱ Refinement layer ʹΑΓɼैདྷख๏ͱൺֱͯ͠΄΅ಉ౳ͷਫ਼౓Ͱ͋Δ͜ͱ͕Θ͔Δɽ
·ͨɼIRL ͱ MixConv Λར༻͢Δ͜ͱʹΑΓɼύϥϝʔλ਺Λ 87.7%࡟‫͍͖ͯͰݮ‬Δ͜ͱ͕֬ೝͰ͖Δɽ࣍ʹɼਤ 3 ʹ֤ख๏ʹΑΔ‫ݦ‬ஶੑϚοϓͷਪఆྫΛࣔ͢ɽ
ਤ 3 ΑΓɼVanilla Ͱ͸‫ؠ‬΍໦ࡐͰฤ·ΕͨҜࢠͳͲΤοδ౳ͷಛ௃͕ີू͢ΔҐஔʹ‫ݦͯͬޡ‬ஶੑ͕‫ݱ‬Ε͍ͯΔɽҰํɼఏҊख๏͸ Refinement layer ͱ Lres ͷಋೖʹΑΓ‫ޡ‬Γͷগͳ͍‫ݦ‬ஶੑϚοϓͱͳ͍ͬͯΔɽ

4.3 ଛࣦؔ਺ͷ༗ޮੑͷௐࠪ
ਤ 4 ʹɼҟͳΔղ૾౓ؒͷը૾Λೖྗͱͨ͠ͱ͖ͷ CCʹΑΔਫ਼౓มԽΛࣔ͢ɽͳ͓ɼ‫ݩ‬ͷղ૾౓Λԣ࣠ͷ 1 ͱ͠ɼӈ΁ҠΔʹͭΕͯղ૾౓͕ 2 ෼ͷ 1 ͣͭ௿Լ͍ͯ͠Δɽਤ
4 ͔ΒɼҰ؏ੑଛࣦΛ༻͍͍ͯͳ͍ Vanilla+R (WithoutLres ) ͱൺ΂ɼVanilla+R+Lres (With Lres ) Ͱ͸௿ղ૾౓Ͱ΋ਫ਼౓͸௿Լ͠ͳ͍ࣄ͕֬ೝͰ͖ΔɽਓؒͷࢹઢͰ͸ɼ32 ෼ͷ 1 ·Ͱͷղ૾౓Ͱ͋Ε͹Ұఆͷ‫ݦ‬ஶੑϚοϓ͕ಘΒΕΔ͜ͱ͔Βɼ֤ղ૾౓ؒͷ੔߹ੑΛߟྀͨ͠Ұ؏ੑଛ

ਤ 4 : ෳ਺ղ૾౓ʹର͢ΔఏҊख๏ͷਫ਼౓ൺֱࣦΛར༻͢Δ͜ͱͰɼਓؒʹ͍ۙࢹ֮తಛ௃ΛֶशͰ͖ͯ
͍Δͱ͍͑Δɽ

5. ͓ΘΓʹຊ‫Ͱڀݚ‬͸ɼ‫ݦ‬ஶੑ༧ଌʹ͓͍ͯ MobileNetV2 ΍ MixConv Λར༻ͨ͠ωοτϫʔΫͷઃ‫ʹܭ‬ΑΓύϥϝʔλ਺Λ໿ 87.7%࡟‫ͨ͠ݮ‬ɽ·ͨɼ֤ղ૾౓ؒͷ‫ݦ‬ஶੑͷ੔߹ੑΛอͭଛࣦؔ਺ Lres ͷఏҊʹΑΓɼLres Λར༻͠ͳ͍৔߹ͱൺ΂ਫ਼౓͕޲্͠ɼैདྷख๏ͱൺ΂ͯύϥϝʔλ਺Λ࡟‫ʹͨ͠ݮ‬΋ؔΘΒͣɼ΄΅ಉ౳ͷਫ਼౓ͱͳΔ͜ͱΛ֬ೝͨ͠ɽࠓ‫ޙ‬͸ɼߋͳΔਫ਼౓޲্Λૂ͏ͨΊʹΞϯαϯϒϧ
Ϟσϧ΍ AutoML ʹΑΔωοτϫʔΫߏ଄ͷ࠷దԽΛߦ͏༧ఆͰ͋Δɽࢀߟจ‫ݙ‬[1] N. Riche, et al., “A multi-scale rarity-based saliency detection with its comparative statistical analysis”, SPIC,
2013.
[2] J. Pan, et al., “Shallow and Deep Convolutional Networks for Saliency Prediction”, CVPR, 2016.
[3] S. Jia, et al., “EML-NET:An Expandable Multi-Layer
NETwork for Saliency Prediction”, IVC, 2020.
[4] T. Judd, et al., “Fixations on low-resolution images.”,
JOV, 2011.

‫੷ۀڀݚ‬[1] ੉ඌढ़‫ و‬౳, “FlowNetC Λಋೖͨ͠ D&T ʹ͓͚Δ෺ମ‫ݕ‬ग़ͷߴਫ਼౓Խ”, ը૾ηϯγϯάγϯϙδ΢Ϝ, 2019.
[2] T. Seo, et al, “Video Object Detection and Tracking
based on Angle Consistency between Motion and Flow”,
IV, 2020.
(ଞ ֶձൃද 1 ݅)