機械知覚&ロボティクスグループ
中部大学 工学研究科 ロボット理工学専攻

2020年度 藤吉研究室 修士論文発表 アブストラクト
Style transfer Test-time Augmentation Domain Adaptation
クラス情報を考慮したスタイル変換によるテストタイム拡張の高精度化に関する研究

今枝 航
Deep Learning Neural Signal Operation Identific

1D Self-Attention Networkによる神経信号からの動作識別に関する研究
田邊 稜
Deep Learning Generative Model Collaborative Learning

Generative Adversarial Networksの共同学習に関する研究
塚原 拓也
Deep Learning Visual Explanation Attention Mechanism

動画像における時空間情報を考慮したDNNによる視覚的説明に関する研究
三津原 将弘

URL: http://mprg.jpΫϥε৘ใΛߟྀͨ͠ελΠϧม‫ʹ׵‬ΑΔςετλΠϜ֦ுͷߴਫ਼౓Խʹؔ͢Δ‫ڀݚ‬ࢦಋ‫ڭ‬तɿ౻٢ ߂࿱

TR190002 ࠓࢬ ߤ
1. ͸͡Ίʹ
ਂ૚ֶशʹΑΔελΠϧม‫׵‬͸ɼೖྗը૾͕࣋ͭελΠϧΛҟͳΔελΠϧ΁ม‫͢׵‬ΔΞϓϩʔνͰ͋ΔɽελΠϧม‫׵‬Λ༻͍ͯҟͳΔυϝΠϯͷը૾ʹม‫͢׵‬Δ͜ͱͰɼ෺ମೝࣝͷυϝΠϯదԠ͕ՄೳͱͳΔɽελΠϧม‫׵‬Λ༻͍ͨυϝΠϯదԠͷ୅දతͳख๏Ͱ͋Δ CyCADA[3] ͸ɼը૾ͷΫϥε͕Ұ؏ͨ͠ม‫׵‬͸ՄೳͰ͋Δ͕ɼೖྗը૾ʹରͯ͠ੜ੒Ͱ͖Δม‫׵‬ը૾͸ 1 ຕͰ͋Δɽຊ‫Ͱڀݚ‬͸ɼελΠϧม‫׵‬ͷࡍʹΫϥε৘ใΛߟྀͯ͠ෳ਺ͷม‫׵‬ը૾Λੜ੒Ͱ͖ΔελΠϧม‫׵‬ख๏ΛఏҊ͢Δɽ͞Βʹɼը૾ͷೝࣝ࣌ʹσʔλ֦ுΛߦ͏ςετλΠϜ֦ுʹఏҊख๏Λ༻͍Δ͜ͱͰߴਫ਼౓Խ͕‫ظ‬଴Ͱ͖ΔɽఏҊख๏͸ɼΫϥε৘ใΛߟྀ͠ͳ͕Β༷ʑͳελΠϧม‫͕׵‬ՄೳͱͳΓɼߴ͍ΞϯαϯϒϧޮՌΛ࣮‫͖Ͱݱ‬Δɽ

Mapping network

Step1: 潜在変数を用いたスタイル変換Step2: 画像の真贋判定

Discriminator
R/F

Generator
Encoder

R/F

Decoder

Step4: 各画像のクラス識別

Class

Style encoder

Step3: 画像の再構成

Sematic
Classifier consistency
Class

Decoder

Encoder

Cycle
consistency

2. ैདྷ‫ڀݚ‬
2.1 ελΠϧม‫׵‬ελΠϧม‫׵‬͸ɼҟͳΔελΠϧΛ࣋ͭը૾ؒʹ͓͍ͯɼҰํ޲΋͘͠͸૒ํ޲ʹը૾ม‫͢׵‬ΔΞϓϩʔνͰ͋Δɽ
ελΠϧม‫׵‬ͷ୅දతͳख๏ͱͯ͠ɼCycleGAN[1] ͕͋ΔɽCycleGAN ͸ɼม‫׵‬ઌͷϖΞը૾Λඞཁͱ͠ͳ͍ม‫׵‬ख๏Ͱ͋Γɼ2 ͭͷελΠϧؒͰ૒ํ޲ʹը૾ม‫͖Ͱ׵‬Δɽ·
ͨɼ3 ͭҎ্ͷελΠϧΛม‫͢׵‬Δख๏ͱͯ͠ɼStarGAN
v2[2] ͕͋ΔɽStarGAN v2 ͸ೖྗը૾ͱ͸ҟͳΔը૾ɼ΋͘͠͸Ψ΢ε෼෍ʹ‫ͮ͘ج‬જࡏม਺͔ΒελΠϧίʔυΛੜ੒͠ɼελΠϧίʔυʹຒΊࠐ·ΕͨελΠϧʹͳΔΑ
͏ʹೖྗը૾Λม‫͢׵‬Δɽ

2.2 υϝΠϯదԠਤ 1 : ఏҊख๏ͷߏ଄

3.2 ‫਺ؔࠩޡ‬‫਺ؔࠩޡ‬͸ɼΫϥεࣝผ‫ث‬Λֶश͢Δ Ltask ͱɼελΠϧม‫͚͓ʹ׵‬ΔΫϥε৘ใͷҰ؏ੑΛߟྀ͢Δ Lsem Λ༻͍Δɽ֤‫਺ؔࠩޡ‬͸ࣜ (1)ɼࣜ (2) ͱఆٛ͢Δɽ

Ltask = CE(C(Genc (xs )), ts )Lsem = CE(C(Genc (G(xs , st )), C(Genc (xs ))+ CE(C(Genc (G(xt , ss )), C(Genc (xt ))

(1)
(2)

υϝΠϯదԠ͸ɼ‫ࢣڭ‬ϥϕϧΛ࣋ͨͳ͍λʔήοτυϝΠϯͱɼ‫ࢣڭ‬ϥϕϧΛ࣋ͭιʔευϝΠϯͷҟͳΔυϝΠϯؒͷΪϟοϓΛܰ‫͢ݮ‬ΔΑ͏ʹֶश͢Δ͜ͱͰɼλʔήοτυϝΠϯͰͷࣝผਫ਼౓Λ޲্ͤ͞Δٕज़Ͱ͋ΔɽελΠϧม‫ʹ׵‬ΑΔυϝΠϯదԠͱͯ͠ Cycle-Consistent
Adversarial Domain Adaptation(CyCADA)[3] ͕ఏҊ͞Ε͍ͯΔɽCyCADA ͸ɼCycleGAN ʹΫϥε৘ใΛߟྀ
͢Δ Semantic consistency Λಋೖ͍ͯ͠Δɽ͜ΕʹΑΓɼιʔευϝΠϯͰࣄલֶशͨࣝ͠ผ‫ث‬Λ༻͍ͯɼม‫׵‬લ‫ޙ‬ͷը૾ʹ͓͚ΔΫϥε৘ใ͕Ұக͢ΔΑ͏ʹֶशͰ͖Δɽ͜͜Ͱɼx ͕ೖྗը૾ɼts ͕ιʔευϝΠϯͷ‫ࢣڭ‬ϥϕϧɼC ͕Ϋϥε෼ྨ‫ث‬ɼG ͕ Generatorɼs ͕ελΠϧίʔυɼCE ͕ΫϩεΤϯτϩϐʔ‫͋Ͱࠩޡ‬Δɽ·ͨɼఴ͑ࣈ͸֤υϝΠϯΛද͓ͯ͠Γɼs ͕ιʔευϝΠϯɼt ͕λʔήοτυϝΠϯΛද͍ͯ͠ΔɽLtask ͸ɼ‫ࢣڭ‬ϥϕϧΛ࣋ͭιʔευϝΠϯͷը૾ͷΈͰ‫ࠩޡ‬Λ‫͢ࢉܭ‬ΔɽLsem ͸ɼΫϥε෼ྨ‫ث‬ΑΓग़ྗ͞ΕΔม‫׵‬લͷΫϥε֬཰ΛԾϥϕϧͱ͠ɼม‫ޙ׵‬ͷΫϥε֬཰ͱͷΫϩεΤϯτϩϐʔ‫ࠩޡ‬Λ‫ܭ‬ࢉ͢Δɽ͜ΕʹΑΓɼը૾ͷม‫׵‬લ‫Ͱޙ‬Ϋϥε৘ใͷҰ؏ੑΛߟֶྀͨ͠श͕ՄೳͱͳΔɽ

3. ఏҊख๏

3.3 ςετλΠϜ֦ுຊ‫Ͱڀݚ‬͸ɼStarGAN v2 Ͱͷը૾ม‫ʹ࣌׵‬ΫϥεࣝผΛߦ͏ิॿλεΫΛ௥Ճ͢Δ͜ͱʹΑΓɼελΠϧม‫࣌׵‬ʹΫϥε৘ใΛߟྀ͢ΔɽิॿλεΫʹ͸ɼCyCADA ͷ
Semantic consistency Λ༻͍Δɽ͜ΕʹΑΓɼຊख๏͸Ϋϥε৘ใΛߟྀ͠ͳ͕Βෳ਺ͷը૾΁ม‫͖Ͱ͕׵‬Δɽຊख๏ΛධՁ࣌ʹσʔλ֦ு͢ΔςετλΠϜ֦ுʹద༻͢Δ
͜ͱͰɼߴਫ਼౓ͳࣝผ͕ՄೳͱͳΔɽ

3.1 Ϋϥε৘ใΛߟྀͨ͠ StarGAN v2

ςετλΠϜ֦ு͸ɼࣝผ࣌ςεταϯϓϧʹରͯ͠σʔλ֦ுΛߦ͍ɼ֤ը૾ʹର͢ΔΫϥε֬཰Λฏ‫ͯ͠ۉ‬࠷ऴతͳ༧ଌ݁ՌΛܾఆ͢Δɽຊख๏Ͱ͸ɼιʔευϝΠϯͰࣄલֶशͨࣝ͠ผ‫΁ث‬λʔήοτυϝΠϯͷը૾Λೖྗ͢ΔࡍʹɼελΠϧม‫ʹ׵‬ΑΔυϝΠϯదԠΛߦ͏ɽಉ࣌ʹɼજࡏม਺Λෳ਺ճαϯϓϦϯά͠ɼ༷ʑͳελΠϧͷը૾΁ม‫͢׵‬Δɽͦͯ͠ɼैདྷͱಉ༷ʹࣜ (3) Ͱఆٛ͢ΔΞϯαϯϒϧਪ࿦ʹͯ࠷ऴతͳ༧ଌ݁ՌΛܾఆ͢Δɽ
NఏҊख๏ͷߏ଄Λਤ 1 ʹࣔ͢ɽఏҊख๏͸ɼιʔεը૾xs ͱλʔήοτը૾ xt ʹରͯ͠ҎԼͷॲཧΛॱʹߦ͏ɽ

Step1 જࡏม਺ z ΑΓੜ੒͞ΕΔελΠϧίʔυ s̃t Λ༻͍ͨελΠϧม‫׵‬ɽ
Step2 ࣮ը૾ xs ͱม‫׵‬ը૾ x̃t ͷਅ‫آ‬൑ఆɽ
Step3 ࣮ը૾ͷελΠϧίʔυ ŝs Λ༻͍ͨը૾ x̃t ͷ࠶ߏ੒ɽ
Step4 ࣮ը૾ xs ͱม‫׵‬ը૾ x̃t ͷΫϥεࣝผɽ
͜͜ͰɼGenerator Λ Encoder-Decoder Ϟσϧͱͨ͠ͱ͖ɼ
ೖྗը૾͕࣋ͭίϯςΩετΛ Encoder Ͱநग़ɼDecoderͰελΠϧ৘ใͷ෇༩Λߦ͏ͨΊɼGenerator ͷதؒಛ௃͸υϝΠϯෆมͷಛ௃ྔͱͳΔɽͦͷͨΊɼGenerator ͷதؒʹΫϥεࣝผ‫ث‬Λ௥Ճ͢Δ͜ͱʹΑΓɼ྆ํͷυϝΠϯʹ͓͍ͯे෼ͳࣝผੑೳ͕‫ظ‬଴Ͱ͖Δɽ·ͨɼ௥Ճͨ͠
Ϋϥε෼ྨ‫ث‬͸ը૾ม‫ʹ࣌׵‬͸༻͍ͳ͍ɽ

pi =

1 !
σi (f (G(x, F (z n ))))
N n=1

(3)

͜͜Ͱɼi ͕Ϋϥε൪߸ɼp ͕༧ଌ݁ՌɼN ͕૿෯਺ɼσ͕ Softmax ؔ਺ɼf ͕ࣄલֶश͞Εͨࣝผ‫ث‬ɼF ͕જࡏม਺ΑΓελΠϧίʔυΛੜ੒͢Δ Mapping networkɼz͕જࡏม਺Ͱ͋Δɽ

4. ධՁ࣮‫ݧ‬ఏҊख๏ͷ༗ޮੑΛࣔͨ͢Ίʹɼม‫׵‬ը૾ͷ܏޲ௐࠪͱ
ςετλΠϜ֦ுʹ͓͚Δࣝผੑೳͷൺֱɼ͓Αͼม‫׵‬ը૾ͷ෼෍ௐࠪΛߦ͏ɽධՁ࣮‫͢༻࢖ʹݧ‬Δσʔληοτ͸ɼ਺ࣈೝࣝ༻ͷσʔληοτͰ͋Δ SVHN ͱ SynthDigitsΛ༻͍Δɽ͜͜ͰɼιʔευϝΠϯʹ͸ɼCG ը૾Ͱ͋Δ
SynthDigitsɼλʔήοτυϝΠϯ͸࣮ը૾Ͱ͋Δ SVHN
ͱ͠ɼΫϥε਺͸ 10 Ͱ͋Δɽ·ͨɼςετλΠϜ֦ுʹ
͓͚Δ૿෯਺͸ 10 ͱ͢Δɽ

ソース
ドメイン
(CG画像)

ターゲット
ドメイン
(実画像)

CycleGANStarGAN v2

提案手法ਤ 2 : ֤ख๏ʹΑΔม‫݁׵‬Ռ(a) ελΠϧม‫׵‬લͷ෼෍(b) ελΠϧม‫ޙ׵‬ͷ෼෍

ਤ 3 : UMAP ʹΑΔ෼෍ͷՄࢹԽ
ද 1 : ςετλΠϜ֦ுʹ͓͚Δࣝผਫ਼౓

4.3 UMAP ʹΑΔม‫׵‬ը૾ͷ෼෍ͷௐࠪ

ResNet-20

CNN(701.4k)(24.3k)

-

74.88

66.54

ϕʔεϥΠϯ

1

77.48

76.44

(StarGAN v2)

10

79.85

79.50

֤υϝΠϯͷը૾ͱఏҊख๏Ͱม‫ͨ͠׵‬ը૾Λ UMAPͰ࣍‫ݩ‬ѹॖ͠ɼՄࢹԽͨ݁͠ՌΛਤ 3 ʹࣔ͢ɽ͜͜Ͱɼ੨
࿮͕ιʔευϝΠϯɼ੺࿮͕λʔήοτυϝΠϯΛදͯ͠
͍Δɽ࣍‫ݩ‬ѹॖʹ࢖༻͢Δը૾ຕ਺͸ɼ֤Ϋϥε͔ΒϥϯμϜʹ 25 ຕબ୒͠ɼ֤υϝΠϯ 250 ຕͱ͢Δɽ·ͨɼѹॖ͢Δಛ௃ʹ͸ఆྔతධՁͰ࢖༻ͨ͠ CNN ͷதؒಛ௃Λ
༻͍Δɽਤ 3(a) ͸ελΠϧม‫׵‬Λߦ͍ͬͯͳ͍ͨΊɼυϝΠϯ͝ͱͷ෼෍͕ҟͳ͍ͬͯΔɽҰํɼελΠϧม‫׵‬Λߦͬͨਤ 3(b) ͸֤Ϋϥεͷ෼෍͕΄΅Ұக͍ͯ͠Δɽ

1

87.73

85.77

5. ͓ΘΓʹ

10

89.21

87.91

ຊ‫Ͱڀݚ‬͸ɼΫϥε৘ใΛߟྀͨ͠ελΠϧม‫׵‬ख๏͓ΑͼɼςετλΠϜ֦ுʹ͓͚ΔελΠϧม‫׵‬Λ༻͍ͨσʔλ֦ுΛఏҊͨ͠ɽධՁ࣮‫͍͓ͯʹݧ‬ɼఏҊख๏ʹΑΔελΠϧม‫׵‬͸ɼΫϥε৘ใΛߟྀͨ͠ม‫͍͖ͯͰ͕׵‬Δ͜ͱΛ֬ೝͨ͠ɽ·ͨɼςετλΠϜ֦ுʹΑΔఆྔతධՁͰ͸ɼม‫׵‬ແ͠ͱൺֱͯ͠ɼ14.33pt ͷਫ਼౓޲্Λ֬ೝͨ͠ɽࠓ‫ޙ‬͸ɼηϚϯςΟοΫηάϝϯςʔγϣϯͳͲͷҟͳΔλεΫ΁ͷదԠΛ‫ݕ‬౼͢Δɽࣝผ཰ [%]
ม‫׵‬ख๏
ม‫׵‬ແ͠ఏҊख๏૿෯਺

4.1 ม‫׵‬ը૾ͷఆੑతධՁ
ఏҊख๏͓Αͼैདྷख๏ (CycleGANɼStarGAN v2) Λ༻͍ͯม‫݁ͨ͠׵‬ՌΛਤ 2 ʹࣔ͢ɽ͜ΕΑΓɼैདྷख๏Ͱ
͸Ϋϥε৘ใͷҡ͕࣋ࠔ೉Ͱ͋Γɼม‫͋Ͱݩ׵‬ΔλʔήοτυϝΠϯͷը૾͕௿ը࣭Ͱ͋ͬͨ৔߹͸ͦͷ܏޲͕‫ݦ‬ஶͰ͋ΔɽҰํɼఏҊख๏ʹΑΔม‫׵‬͸ɼΫϥε৘ใΛҡ࣋ͨ͠ঢ়ଶͰҟͳΔελΠϧͷը૾΁ม‫͍͖ͯͰ׵‬Δ͜ͱ͕
෼͔Δɽ

4.2 ςετλΠϜ֦ுΛ༻͍ͨఆྔతධՁ
ςετλΠϜ֦ுʹ࢖༻͢ΔࣄલֶशࡁΈࣝผ‫ʹث‬͸ɼ
ResNet-20 ͱɼυϝΠϯؒͷࠩҟΛ‫ٵ‬ऩ͢Δ͜ͱ͕ࠔ೉ͳ
5 ૚ͷ CNN Λ༻͍ͯධՁ࣮‫ݧ‬Λߦ͏ɽςετλΠϜ֦ுʹ
ΑΔࣝผਫ਼౓Λද 1 ʹࣔ͢ɽ͜͜Ͱɼ֤ωοτϫʔΫͷύϥϝʔλ਺͸Ϟσϧ໊ͷԼʹࣔ͢ɽද 1 ΑΓɼResNet-20 ͷ݁Ռʹ͓͍ͯɼఏҊख๏͸ม‫׵‬ແ͠ͱൺֱͯࣝ͠ผਫ਼౓ͷ޲্͕֬ೝͰ͖Δɽ·ͨɼ૿෯਺ 1 Ͱ͸ɼఏҊख๏ʹΑΔࣝ
ผਫ਼౓͸ϕʔεϥΠϯͷ StarGAN v2 ͱൺֱ͠ɼ10.25pt
ͷਫ਼౓޲্Λ֬ೝͨ͠ɽ͞Βʹɼ૿෯਺ 10 Ͱ͸ɼ૿෯਺
1 ͱൺֱͯ͠ 1.48pt ͷਫ਼౓޲্Λ֬ೝͨ͠ɽ͜ͷ܏޲͸
CNN ͷࣝผ݁Ռʹ͓͍ͯ΋ಉ༷Ͱ͋Δɽࢀߟจ‫ݙ‬[1] J.Y.Zhuɼet al.ɼ“Unpaired Image-to-Image Translation
using Cycle-Consistent Adversarial Networks”In ICCVɼ2017ɽ
[2] Y.Choiɼet al.ɼ“StarGAN v2: Diverse Image Synthesis
for Multiple Domains”ɼIn CVPRɼ2020ɽ
[3] J.Hoffmanɼet al.ɼ“CyCADA: Cycle Consistent Adversarial Domain Adaptation”ɼIn ICMLɼ2018ɽ‫੷ۀڀݚ‬[1] ࠓࢬߤ ౳ɼ“Attention ‫ߏػ‬Λಋೖͨ͠ CycleGAN ʹΑΔࣝผʹ༗ޮͳελΠϧม‫”׵‬, ը૾ͷೝࣝɾཧղγϯϙδ΢Ϝ,2019.
[2] ࠓࢬߤ ౳ɼ“Generative Adversarial Networks Λ༻͍͔ͨΒ͋͛ϐοΩϯάʹ͓͚Δલॲཧͱͯ͠ͷը૾ม‫”׵‬, Ϗδϣϯٕज़ͷ࣮ར༻ϫʔΫγϣοϓ, 2020.

1D Self-Attention Network ʹΑΔਆ‫ܦ‬৴߸͔Βͷಈ࡞ࣝผʹؔ͢Δ‫ڀݚ‬ࢦಋ‫ڭ‬तɿ౻٢߂࿱
TR19005 ాᬒɹྏ1. ͸͡Ίʹ
ਓ෺ͷಈ࡞ࣝผ͸ɼը૾ϕʔεͷࣝผ๏ [1] ͱ৴߸ϕʔεͷࣝผ๏ [2] ʹେผͰ͖Δɽը૾ϕʔεͷख๏͸ɼRGB Χϝϥ͔Βऔಘͨ͠ը૾Λ༻͍ΔͨΊɼ૷ண‫ܭܕ‬ଌσόΠεΛඞཁͱͤͣɼඇ઀৮Ͱಈ࡞ͷࣝผ͕ՄೳͰ͋Δɽ͔͠͠ɼ
෺Λ௫ΉͳͲͷ෺ମʹରͯ͠खઌͷΈͰߦ͏ಈ࡞͸ɼΦΫϧʔδϣϯ͕ൃੜ͢Δͱࣝผ͕ࠔ೉ͱͳΔɽҰํɼ৴߸ϕʔεͷख๏͸ɼਓମʹ૷ணͨ͠σόΠε͔Β‫ిے‬৴߸Λ༻͍ΔͨΊɼΦΫϧʔδϣϯ͕ൃੜ͠ͳ͍ͱ͍͏ϝϦοτ͕͋Δɽखઌͷಈ࡞ࣝผΛߦ͏ͨΊʹ͸ɼखઌͷಈ࡞ʹਂؔ͘
ΘΔ৴߸Λ‫ܭ‬ଌ͢Δඞཁ͕͋Δɽͦ͜Ͱຊ‫Ͱڀݚ‬͸ɼਓମͷखटʹҐஔ͠ɼखͷ‫ے‬೑Λࢧ഑͢Δखࠜ؅ਆ‫͔ܦ‬Β‫ܭ‬ଌՄೳͳਆ‫ܦ‬৴߸ʹண໨͢Δɽਆ‫ܦ‬৴߸ͷ࣌‫ྻܥ‬σʔλ͔Βಈ࡞‫ݻ‬༗ͷಛ௃Λଊ͑ΔͨΊʹɼ1D-Self-Attention BlockΛఏҊ͢Δɽ

⨀Pointwise conv
1×1×C’

1×1×C

H

Self attention map
1×1×C’

C

& % $

W
H×W×C

,"
1×1×C

1×1×C

Generated SAM
1×W×C’⨀

Pointwise conv
input

Pointwise conv

1×1×C’

1×1×C

& % $
1×W×C

Bn/Relu⨁

Bn/Relu

⨁

,"
1×1×C

Output

SAM
1×W×C

1×1×C

スキップ機構: 近傍時刻１(t - 1)

: 注目時刻( t ): 近傍時刻２(t + 1)

ਤ 1 : 1D Self-Attention Block ͷߏ଄

2. ैདྷ‫ڀݚ‬৴߸ϕʔεͷैདྷख๏ͱͯ͠ɼMorbidoni Β͸ද໘‫ిے‬ਤ (sEMG) Λ֊૚‫ܕ‬χϡʔϥϧωοτϫʔΫʹೖྗͯ͠ɼ
ਓ෺ͷελϯεϑΣʔζͱεΠϯάϑΣʔζͷಈ࡞ࣝผΛߦ͏ख๏ΛఏҊ͍ͯ͠Δ [3]ɽ·ͨɼOzal Β͸ਓؒͷ৺ిਤ͔Βϑϥάϝϯτͱ‫ݺ‬͹ΕΔখྖҬΛαϯϓϦϯά͠ɼ
CNN ΛదԠ͢Δ͜ͱͰ 17 Ϋϥεͷ৺ෆશΛ‫ݕ‬ग़͢Δख๏ΛఏҊ͍ͯ͠Δ [2]ɽ͜ΕΒͷैདྷख๏͸ɼΫϥεࣝผͷͨΊͷಛ௃நग़ʹଟ૚ύʔηϓτϩϯ΍৞ΈࠐΈχϡʔϥϧωοτϫʔΫΛ༻͍͍ͯΔɽ͔͠͠ɼ৞ΈࠐΈ૚͸ɼۙ๣ͷ࣌ࠁʹண໨͢ΔͨΊɼ཭Εͨ࣌ࠁͷॏཁͳมԽΛଊ͑Δ
͜ͱʹద͍ͯ͠ͳ͍ɽ

3. ఏҊख๏

3.3. ωοτϫʔΫͷߏ଄ຊ‫Ͱڀݚ‬͸ɼ̏૚ͷ 1D-SAB ͔ΒͳΔ 1D-SAN Λ༻͍ΔɽҰ૚໨ͷ 1D-SAB ʹ͓͚Δۙ๣࣌ࠁͷִؒΛ 3 ࣌ࠁɼ
2ɼ3 ૚໨ͷ 1D-SAB ʹ͓͚Δۙ๣࣌ࠁͷִؒΛ 5 ࣌ࠁͱ͢Δɽ͜ΕʹΑΓɼ཭Εͨ࣌ࠁͷ৴߸ͱͷมԽΛߟྀͰ͖Δɽ
1D-SAN ͷߏ଄ͱ֤ 1D-SAB ͷۙ๣ઃఆΛਤ 2 ʹࣔ͢ɽ

t-3

input

t+3

3.1. खࠜ؅ਆ‫ܦ‬ຊ‫Ͱڀݚ‬͸ɼਓ෺ͷखઌͷಈ࡞Λର৅ͱ͢ΔͨΊɼਓମͷखटʹҐஔ͠ɼखͷ‫ے‬೑Λࢧ഑͢Δखࠜ؅ਆ‫͔ܦ‬Β‫ܭ‬ଌՄೳͳਆ‫ܦ‬৴߸Λ༻͍Δɽखࠜ؅ਆ‫ʹܦ‬͸ɼᒷࠎਆ‫ܦ‬(Radial narve)ɼਖ਼தਆ‫( ܦ‬Median narve)ɼईࠎਆ‫( ܦ‬Ulnar narve) ͕͋Δɽ͜ΕΒͷਆ‫ܦ‬৴߸Λ Mudra wearable
device ʹΑΓ‫ܭ‬ଌ͢ΔɽαϯϓϦϯάִؒ͸ 1ms ͱ͢Δɽ

3.2.1D Self-Attention Networks(1D-SAN)ຊ‫Ͱڀݚ‬͸ɼSelf-Attention Networks(SAN)[4] Ͱ༻͍ΒΕΔ Self-Attention Block(SAB) Λ 1 ࣍‫ݩ‬σʔλʹରԠͤͨ͞ɼ1D-SAB ΛఏҊ͠ɼ1D-SAN Λߏ੒͢ΔɽఏҊ
͢Δ 1D-SAB ͷߏ଄Λਤ 1 ʹࣔ͢ɽ1D-SAB ʹ͸ɼखࠜ؅ਆ‫͔ܦ‬Βऔಘͨ࣌͠‫ྻܥ‬৴߸Λೖྗ͢Δɽೖྗͨ͠৴߸
͸ɼ̍࣌ࠁ͝ͱʹॲཧ͠ɼରԠ͢Δ࣌ࠁͷ Self-AttentionΛࢉग़͢Δɽਤ 1 ʹ͓͚Δ੨৭ͷ஋Λॲཧͷ஫໨࣌ࠁͱ͠
ͨ࣌ɼ੺৭Λۙ๣࣌ࠁ̍, ྘৭Λۙ๣࣌ࠁ̎ͱ͢Δɼ֤ۙ๣࣌ࠁʹରͯ͠ɼ஫໨࣌ࠁ (੨) ͱͷ Pointwise conv ॲཧΛߦ͏ɽ·ͨɼ஫໨࣌ࠁͱۙ๣࣌ࠁΛɼП͓ΑͼСͷֶशՄೳͳؔ਺ (1D-CNN) ΁ೖྗ͠ɼؔ܎ؔ਺ЎΛ‫ٻ‬ΊΔɽؔ܎ؔ਺Ўͷग़ྗ͸ɼؔ਺Пͷग़ྗ͔Βؔ਺Сͷग़ྗΛ‫ࢉݮ‬ͨ͠΋ͷͰ͋ΓɼϚοϐϯάؔ਺ЍʹΑͬͯ̍ͭ໨ͷॲཧͷग़ྗͱνϟϯωϧ਺Λ߹ΘͤΔɽͦͷ‫ޙ‬ɼ2 ͭͷग़ྗͷཁૉੵΛࢉग़͢Δɽ͜ͷॲཧΛۙ๣࣌ࠁ෼ߦ͍૯࿨͢Δɽ
ੜ੒ͨ͠ Self-Attention Map(SAM) ͸ Pointwise conv ॲཧʹΑΓೖྗνϟωϧ਺ͱಉ͡νϟωϧʹ͢Δɽ͜ͷग़ྗʹɼεςοϓ‫ͯ͠ͱߏػ‬ೖྗ৴߸ΛՃࢉ͠ɼ࠷ऴग़ྗͱ͢
Δɽ1D-SAB Λ༻͍Δ͜ͱͰɼਆ‫ܦ‬৴߸಺ͷॏཁͳҐஔʹେ͖ͳॏΈΛ༩͑Δ͜ͱ͕Ͱ͖ɼ཭Εͨ৴߸஋ͱͷؔ܎ੑΛߟྀͰ͖Δɽ

t-5

1D-SAB1ਓ෺ͷखઌͷಈ࡞͸ɼͦͷछྨʹΑΓಈ࡞͕࣌ؒҟͳΔɽͦͷͨΊɼಈ࡞ͷ୹‫ظ‬త͓Αͼ௕‫ظ‬తͳมԽΛଊ͑Δࣄ͕ॏ
ཁͱͳΔɽຊ‫Ͱڀݚ‬͸ɼ࣌‫ྻܥ‬σʔλʹ͓͍ͯɼपล࣌ࠁΛߟྀͨ͠ಛ௃நग़͕Մೳͳ 1D-Self-Attention Block(1DSAB) ʹΑΓߏ੒͞ΕΔ 1D-Self-Attention Networks(1DSAN) Λ༻͍ͨಈ࡞ࣝผ๏ΛఏҊ͢Δɽ

FC

t

t

1D-SAB2

1D-SAB3
・
・
・
・
・

t
t-5

t+5

・
・
・
・
・

t+5

・
・
・

6 class

ਤ 2 : ωοτϫʔΫߏ଄ͱۙ๣

3.4. ಈ࡞ࣝผͷྲྀΕຊ‫Ͱڀݚ‬ͷಈ࡞ࣝผ͸ɼ3 ͭͷ Step ͱͳΔɽStep 1 ͸ɼ
Mudra wearable device Λ༻͍ͯऔಘͨ͠ਆ‫ܦ‬৴߸ΛҰఆ۠ؒ͝ͱʹ‫ݻ‬ఆ௕ͰαϯϓϦϯά͠ɼ1D-SAN Ͱਪ࿦͕Ͱ
͖Δσʔλ΁੔‫͢ܗ‬ΔɽStep 2 ͸ɼֶशࡁΈ 1D-SAN ʹɼࣝผର৅ͷσʔλͷ 0 ࣌ࠁΛ‫ͯ͠ͱ఺ى‬ɼೖྗ͢ΔσʔλͷྖҬΛ 10 ࣌ࠁͣͭ‫܁‬Γฦ͠ೖྗ͠ɼಈ࡞ࣝผ͢ΔɽStep
3 ͸ɼ֤۠ؒͷಈ࡞ࣝผ݁ՌΛ౷߹ͯ͠࠷ऴతͳಈ࡞ࣝผ
݁ՌΛग़ྗ͢Δɽ

Step 1ɿલॲཧຊ‫Ͱڀݚ‬ѻ͏ਆ‫ܦ‬৴߸೾‫ܗ‬͸ɼՄม௕ͷσʔλͰ͋Δɽ
1D-SAN Ͱਪ࿦Λߦ͏ͨΊʹɼ64 ࣌ࠁ෼ͷ 1 ࣍‫ݩ‬೾‫ܗ‬ΛαϯϓϦϯά͢Δɽ

Step 2ɿαϯϓϦϯά࣌ࠁͰͷಈ࡞ࣝผಈ࡞ࣝผ͸ɼ10 ࣌ࠁͣͭαϯϓϦϯά۠ؒΛͣΒ͠ͳ͕Β 1D-SAN ʹೖྗ͢Δɽ

Step 3ɿಈ࡞ࣝผ݁Ռͷ౷߹֤࣌ࠁʹ͓͚ΔωοτϫʔΫͷग़ྗΛΫϥεຖʹྦྷੵ͢
Δɽྦྷੵ஋͕࠷େͷΫϥεΛಈ࡞ࣝผ݁Ռͱ͢Δɽࣝผ݁ՌྦྷੵॲཧΛਤ 3 ʹࣔ͢ɽ

4. ධՁ࣮‫ݧ‬ఏҊख๏ͷ༗ޮੑΛࣔͨ͢Ίʹɼ1 ࣍‫ݩ‬೾‫ʹܗ‬ରԠͨ͠৞ΈࠐΈχϡʔϥϧωοτϫʔΫ (1D-CNN) ͱೝࣝਫ਼౓
ͷൺֱΛߦ͏ɽ·ͨɼϑʔϦΤม‫͠׵‬ɼप೾਺ۭؒΛೖྗͱͨ͠ࡍͷਫ਼౓ͱ΋ൺֱ͢Δɽ

4.1. σʔληοτຊ‫Ͱڀݚ‬͸ɼ6 Ϋϥεͷಈ࡞ʹର͢Δਆ‫ܦ‬৴߸Λ 100 ճ‫ܭ‬ଌͨ͠ 600 αϯϓϧΛ༻͍Δɼ͜ΕΒΛ 6:4 Ͱֶश͓ΑͼධՁσʔλʹ෼ׂͯ͠༻͍ΔɽΫϥεͷσʔλऔಘ࣌ͷ

・・・

t=1

t=2

input

1D-SAN

t=n

1D-SAN

output・・・

1D-SAN

6クラス分の識別結果[!!!, !$!, !%!, !&!, !'!, !(!],[!!$, !$$, !%$, !&$, !'$, !($],

要素和・・・ [!!#, !$#, !%#, !&#, !'#, !(#]

͸‫ݮ‬গͨ͠ɽҰํͰఏҊख๏͸ɼPush Λ Twist ͱ‫ࣝޡ‬ผ͢Δ͜ͱ͕ଟ͍ɽPush ʹରͯ͠ఏҊख๏ͷਫ਼౓͕௿Լ͠
ͨཁҼ͸ɼ෺Λԡ͠ࠐΉಈ࡞Λ͢ΔࡍʹɼྗΛࠐΊͯ‫͘ڧ‬ԡ͠ࠐΉέʔε͕͋ΓɼPush Ͱ͋Γͳ͕Β Twist ͷΑ͏ʹৼಈ͢Δ৴߸͕‫ܭ‬ଌ͞Εͨ͜ͱ͕‫ݪ‬Ҽͱߟ͑ΒΕΔɽ͜
ͷΑ͏ͳέʔεΛֶशσʔλʹ΋௥Ճ͢Δඞཁ͕͋Δɽ·ͨɼTurn ΍ Twist ͳͲɼ࣌‫ྻܥ‬తͳมԽ͕େ͖ͳΫϥεʹର͢Δਫ਼౓͕޲্͍ͯ͠Δ͜ͱ͔ΒɼఏҊख๏͸৴߸஋͕࣌‫ྻܥ‬తʹେ͖͘มԽ͢ΔΫϥεʹ༗ޮͰ͋Δɽ

最終識別結果argmax( ∑ #" "!", ∑ #" "$", ∑ #" "%", ∑ #" "&" , ∑ #" "'" , ∑ #" "(" )ද 1 : Confusion Matrix ʹΑΔࣝผ݁Ռͷൺֱ

ਤ 3 : ࠷ऴॲཧ൑ఆ

ը૾Λਤ 4 ʹࣔ͢ɽσʔλ͸ɼMudra wearable device Λඃ‫ऀݧ‬ͷӈखटʹ૷ண͠ɼಈ࡞தͷ 1 ࣍‫ݩ‬೾‫ܗ‬Λऩूͨ͠ɽ
ਤ 5 ʹಈ࡞ Push ʹର͢Δਆ‫ܦ‬৴߸Λࣔ͢ɽ

Push

Twist

Take

Type

ද 2 ʹɼप೾਺ྖҬʹ͓͚Δ֤ख๏ͷਫ਼౓Λࣔ͢ɽఏҊख๏ͱ 1D-CNN ͰൺֱΛߦͬͨ݁Ռɼप೾਺ྖҬͰ͸ɼ
1D-SAN ͷಈ࡞ࣝผਫ਼౓͕ 80.0 ˋɼ1D-CNN ͷಈ࡞ࣝผਫ਼౓͕ 90.4 ˋͱͳΓɼ1D-SAN ͷਫ਼౓͕ԼճͬͨɽҰํͰϑʔϦΤม‫Ͱ͠ͳ׵‬͸ɼ1D-SAN ͷࣝผਫ਼౓͸ 91.2 ˋͱ޲্ͨ͠ɽ1D-SAN ͸ɼ૬ରతͳҐஔؔ܎ͱͦͷॏཁੑ
Λ Self-Attention ʹΑΓද‫͖Ͱݱ‬ΔͨΊϑʔϦΤม‫͠ͳ׵‬Ͱಈ࡞ࣝผ͕ՄೳͰ͋Δɽ
ද 2 : प೾਺ྖҬʹ͓͚Δਫ਼౓ൺֱ

Turn

Put

ਤ 4 : ର৅ಈ࡞ΫϥεωοτϫʔΫ

ϑʔϦΤม‫͋׵‬Γ (%)ϑʔϦΤม‫( ͠ͳ׵‬%)1D-SAN

80.0

91.2

1D-CNN

90.4

90.4

5. ͓ΘΓʹ
ຊ‫Ͱڀݚ‬͸ɼखઌͷ‫ے‬೑Λࢧ഑͢Δखࠜ؅ਆ‫͔ܦ‬Βऔಘͨ͠ਆ‫ܦ‬৴߸͔Βͷಈ࡞ࣝผΛߦ͏ͨΊʹɼ1 ࣍‫ݩ‬೾‫ʹܗ‬ରԠͨ͠ 1D Self-Attention Network ΛఏҊͨ͠ɽਫ਼౓ൺ
ֱ࣮‫Ͱݧ‬͸ɼैདྷख๏ͷ 1D-CNN ʹରͯ͠ɼఏҊख๏͕
໿ 0.8 ϙΠϯτ্ճΓɼಈ࡞ࣝผʹ͓͍ͯɼఏҊख๏͕༗
ޮͰ͋Δ͜ͱΛࣔͨ͠ɽ·ͨɼPush ͷΑ͏ͳਆ‫ܦ‬৴߸஋͕࿈ଓతʹେ͖͘ͳΔΫϥεͰ͸ɼਫ਼౓͕௿Լ͕ͨ͠ɼTurn
΍ Twist ͳͲɼ࣌‫ྻܥ‬తͳมԽ͕ଊ͑΍͍͢Ϋϥεʹؔͯ͠ਫ਼౓͕޲্ͨ͠ɽ

ࢀߟจ‫ݙ‬

ਤ 5 : ৴߸σʔλαϯϓϧ (Push)

4.2. ࣮‫֓ݧ‬ཁ
1D-CNN Λ༻͍ͨϞσϧͱ 1D-SAB Λ༻͍ͨϞσϧͷਫ਼౓ൺֱΛߦ͏ɽ1D-CNN ͸ɼ2 ૚ͷ৞ΈࠐΈ૚ͱग़ྗ૚ͷ‫ ܭ‬3 ૚ͷߏ଄Ͱ͋Δɽֶश͸ɼEpoch ਺Λ 20000ɼ
BatchSize Λ 128ɼαϯϓϦϯά෯Λ 64 ͱ͢ΔɽOptimizer͸ Momentum SGD ͱ͠ɼֶश཰Λ 0.001 ɼLoss ؔ਺ʹ
CrossEntropy Loss Λ༻͍Δɽ

4.3. ࣮‫݁ݧ‬Ռ

ද 1 ʹ֤ख๏ͷ Confusion Matrix Λࣔ͢ɽൺֱ࣮‫ݧ‬ͷ݁Ռɼ1D-CNN ͷಈ࡞ࣝผਫ਼౓͸ 90.4 ˋɼఏҊख๏͸
91.2 ˋͱͳΓɼఏҊख๏͕ಈ࡞ࣝผͰ༗ޮͰ͋Δ͜ͱ͕Θ͔ͬͨɽఏҊख๏͸ɼTurnɼTwistɼType ͷਫ਼౓͕޲্
͠ɼTakeɼPush ͷਫ਼౓͕௿Լͨ͠ɽ1D-CNN Ͱ͸ɼTwistΛ Turn ͱ‫ࣝޡ‬ผ͕ͨ͠ɼఏҊख๏Ͱ͸ɼTurn ΁ͷ‫ࣝޡ‬ผ

[1] T. Xiao,et al., “Reasoning About Human-Object
Interactions Through Dual Attention Networks”,ICCV, 2019.
[2] O. Yildirim, et. al. , “Arrhythmia detection using
deep convolutional neural network with long duration ECG signals”, Computers in Biology andMedicine, vol102, 2018, p411 420.
[3] C.Moribidoni,et al., “A Deep Learning Approachto EMG-Based Classification of Gait Phases during
Level Ground Walking” ,MDPI, 2019.
[4] Hengshuang.Zhao,et al., “Exploring Self attention
for Image Recognition” ,CVPR, 2020.

‫੷ۀڀݚ‬[1] ాᬒྏ ౳, “Self-Attention Networks ʹΑΔਆ‫ܦ‬৴߸͔Βͷಈ࡞ࣝผ”, Ϗδϣϯٕज़ͷ࣮ར༻ϫʔΫγϣοϓ, 2020.

Generative Adversarial Networks ͷ‫ڞ‬ಉֶशʹؔ͢Δ‫ڀݚ‬ࢦಋ‫ڭ‬तɿ౻٢߂࿱

TR19006 ௩‫୓ݪ‬໵
1. ͸͡Ίʹ
Generative Adversarial NetworksʢGANsʣ͸ Generator ͓Αͼ Discriminator Λఢରతʹֶश͢Δ͜ͱͰɼ࣮
ࡏ͠ͳ͍ը૾Λੜ੒͢Δख๏Ͱ͋ΔɽGenerator ͸ɼજࡏม਺Λ༻͍ͯը૾Λੜ੒͠ɼੜ੒ը૾͕ Discriminator ʹΑΓ࣮ը૾ͱࣝผ͞ΕΔΑ͏ʹֶश͢ΔɽDiscriminator͸ɼ࣮ը૾ͱੜ੒ը૾ΛࣝผͰ͖ΔΑ͏ʹֶश͢ΔɽGANs
ͷֶश࣌ɼGenerator ͷੜ੒ը૾ʹภΓ͕ੜ͡ΔͱɼDiscriminator ͸࣮ը૾ͱੜ੒ը૾Λ؆୯ʹࣝผͰ͖ɼϞʔυ่յΛ‫͕͋ͱ͜͢͜ى‬Δɽ͜ͷ໰୊Λղܾ͢ΔͨΊʹɼෳ਺ͷ Generator Λ༻͍Δ͜ͱͰɼϞʔυ่յͷൃੜΛ཈੍
͢Δख๏͕ఏҊ͞Ε͍ͯΔ [2]ɽ͔͠͠ɼ͜ͷख๏͸ Generator ͕ಠཱͯ͠ը૾Λੜ੒͢ΔͨΊɼੜ੒ը૾ʹภΓ͕ੜ͡Δ໰୊͸ղܾ͍ͯ͠ͳ͍ɽͦ͜Ͱɼຊ‫Ͱڀݚ‬͸ੜ੒ը૾ͷภΓΛ཈੍͢ΔͨΊʹɼෳ਺ͷ Generator ͕஌ࣝΛసҠ͠ͳ͕Β‫ڞ‬ಉֶशͯ͠ը૾Λੜ੒͢Δख๏ΛఏҊ͢Δɽ

2. ؔ࿈‫ڀݚ‬ैདྷख๏Ͱ͋Δ Deep Convolutional GANs ͓Αͼ MultiAgent Diverse GANs ʹ͍ͭͯड़΂Δɽ
Deep Convolutional GANsʢDCGANsʣ[1] ͸ɼ৞ΈࠐΈ૚Λ༻͍Δ͜ͱͰੜ੒ը૾ͷ࣭Λ޲্ͤͨ͞ GANs
ͷख๏Ͱ͋Δɽ࣮ը૾ͱͯ͠ਖ਼໘ͷ‫إ‬ը૾ͷΈͷσʔλ౳ͷྨࣅͨ͠σʔλͷΈѻ͏৔߹ʹ͓͍ͯ͸ɼ࣮ը૾ͱ۠ผ
͕Ͱ͖ͳ͍઱໌ͳը૾Λੜ੒Ͱ͖Δɽֶश͢ΔࡍʹϞʔυ่յ͕ൃੜ͢Δ৔߹͕ଟ͘ɼ҆ఆֶͨ͠श͕ࠔ೉Ͱ͋Δɽ
Multi-Agent Diverse GANsʢMAD-GANsʣ[2] ͸ɼෳ਺ͷ Generator ͓Αͼ̍ͭͷ Discriminator Ͱߏ੒͞Εͨ GANs ͷख๏Ͱ͋ΔɽDiscriminator ͸࣮ը૾ͱੜ੒ը
૾Λࣝผ͢ΔͷΈͳΒͣɼͲͷ Generator ͔Βੜ੒͞Εͨੜ੒ը૾Ͱ͋Δ͔Λࣝผ͢ΔΑ͏ʹֶश͢ΔɽMAD-GANs
͸ɼ̍ͭ Generator ͕ੜ੒͢Δը૾ʹภΓ͕ੜͨ͡৔߹ɼ
ଞͷ Generator ͸ภͬͨը૾ͱ͸ҟͳΔը૾Λੜ੒͓ͯ͠ΓɼDiscriminator ͸࣮ը૾ͱੜ੒ը૾Λ؆୯ʹ͸ࣝผͰ͖ͳ͍ɽ͜ΕʹΑΓɼMAD-GANs Λ༻͍Δ͜ͱͰϞʔυ่յͷൃੜΛ཈੍͢Δ͜ͱ͕ՄೳͱͳΔɽ͔͠͠ɼੜ੒ը
૾ͷภΓ͕ੜ͡Δ͜ͱࣗମΛ๷͙͜ͱ͸Ͱ͖ͳ͍ɽ

3. ఏҊख๏
ຊ‫Ͱڀݚ‬͸ෳ਺ͷ Generator Λ༻ֶ͍ͯशΛߦ͍ɼGenerator ͕‫͍ޓ‬ͷ஌ࣝΛసҠ͠ͳ͕Β‫ڞ‬ಉֶशͯ͠ը૾ੜ੒
Λߦ͏ GANs ͷख๏ΛఏҊ͢Δɽ

3.1 GANs ͷ‫ڞ‬ಉֶशఏҊख๏ͷωοτϫʔΫߏ଄Λਤ 1 ʹࣔ͢ɽֶशʹ࢖
༻͢Δ Generator ͷ਺Λ k ͱ͢Δͱɼ֤ Generator ͸
G1 , . . . , Gk Ͱද͞ΕΔɽ֤ Generator ͸ɼજࡏม਺ z ∼ Pz
Λ༻͍ͯੜ੒ը૾Λੜ੒͢ΔɽͦͷͨΊɼk ຕͷੜ੒ը૾͕
ಘΒΕΔɽ·ͨɼ֤ Generator ͷಛ௃Ϛοϓ·ͨ͸ੜ੒ը૾Λ஌ࣝసҠάϥϑΛ༻͍ͯసҠͤ͞Δ͜ͱͰɼGeneratorಉ࢜Ͱͷ஌ࣝͷ఻ୡΛՄೳʹ͍ͯ͠ΔɽDiscriminator ͸
D Ͱද͞Εɼ࣮ը૾ͱੜ੒ը૾Λࣝผ͢ΔͷΈͳΒͣɼͲͷ Generator ͔Βੜ੒͞Εͨੜ੒ը૾Ͱ͋Δ͔Λࣝผ͢ΔɽͦͷͨΊɼ࠷ऴ૚ͷग़ྗ஋ͷϢχοτͷ਺͸ k + 1 ‫ݸ‬ͱͳ͓ͬͯΓɼιϑτϚοΫεؔ਺Λ༻͍ͯΫϥεείΞ
d1 , . . . , dk+1 Λग़ྗ͢ΔɽείΞ d1 , . . . , dk ͸֤ Generator ͔Βੜ੒͞Εͨੜ੒ը૾Ͱ͋Δ֬཰Λද͠ɼείΞdk+1 ͸࣮ը૾Ͱ͋Δ֬཰Λද͢ɽωοτϫʔΫ಺෦ʹ͸ɼֶशͷ҆ఆԽΛਤΔͨΊɼBatch Normalization ͕༻͍Β
ΕΔ৔߹͕ଟ͍ɽ͔͠͠ɼGANs ʹ͓͚Δֶशͷ҆ఆԽʹ͸ Batch Normalization ͸ෆे෼Ͱ͋Δɽͦ͜Ͱɼຊख๏
Ͱ͸ Discriminator ʹ Spectral NormalizationʢSNʣΛಋೖ͢Δɽ͜ΕʹΑΓɼଛࣦؔ਺ʹมߋΛՃ͑Δ͜ͱͳ͘Ϟʔυ่յΛ཈੍͠ɼGANs ʹ͓͚Δֶशͷ҆ఆԽΛਤΔ͜ͱ͕ՄೳͱͳΔɽਤ 1: ఏҊख๏ͷωοτϫʔΫߏ଄

3.2 ஌ࣝసҠάϥϑʹΑΔ࠷దԽఏҊख๏͸ɼ஌ࣝసҠάϥϑΛಋೖ͢Δ͜ͱͰɼ֤ Generator ͕‫ʹ͍ޓ‬஌ࣝసҠ͠ͳ͕Βֶश͢Δɽ̏ͭͷ Generator Λ༻͍ͨ৔߹ʹ͓͚Δ஌ࣝసҠάϥϑΛಋೖͨ͠ఏ
Ҋख๏Λਤ 2 ʹࣔ͢ɽֶशʹ࢖༻͢Δ Generator ͷ਺Λ kͱ͢Δͱɼ֤ Generator ͸ G1 , . . . , Gk ɼDiscriminator ͸D ͱͳΓɼϊʔυͱͯ͠ද͢ɽ֤ Generator ͷϊʔυؒʹ͸̎ͭͷΤοδΛఆٛ͠ɼͦΕͧΕҟͳΔ޲͖ͷ༗޲άϥϑͰද͢ɽ֤Τοδͷ޲͖͸ɼֶश࣌ʹޯ഑ͷ৘ใ͕఻ΘΔํ޲Λද͍ͯ͠Δɽ֤ Generator ͷϊʔυؒͷΤοδ͸
Generator ಉ࢜ͷ஌ࣝͷ఻ୡΛද͠ɼi ൪໨ͷ GeneratorͷΤοδ LD,i ͸ i ൪໨ͷ Generator ͕ੜ੒ͨ͠ੜ੒ը૾͕
Discriminator ʹ࣮ը૾ͱ൑அ͞Ε͔ͨͲ͏͔ͷ৘ใͷ఻ୡΛද͢ɽ·ͨɼΤοδ Ladversarial ͸ɼDiscriminator ͷࣝผ݁Ռ͕ਖ਼͍͔͠Ͳ͏͔ͷ৘ใͷ఻ୡΛද͢ɽ֤Τοδʹ͸‫ݸ‬ผͷଛࣦؔ਺Λఆٛ͠ɼ֤ Generator ͷํ޲Λ޲͍ͨΤοδͷଛࣦؔ਺ʹ͸ήʔτؔ਺Λಋೖ͢Δ͜ͱͰɼ఻
೻͞ΕΔଛࣦΛ੍‫͢ޚ‬Δɽຊख๏Ͱ͸ɼήʔτؔ਺ͱͯ͠
Through GateɼCutoff GateɼNegative linear Gate ͓Αͼ Positive linear Gate Λ༻͍ΔɽThrough Gate ͸ɼೖྗ͞Εͨαϯϓϧ͝ͱͷଛࣦΛͦͷ··௨ؔ͢਺Ͱ͋Δɽ
Cutoff Gate ͸ଛࣦΛ 0 ͱ͠ɼଛࣦͷ‫ࢉܭ‬ΛߦΘͳ͍ؔ਺Ͱ͋Γɼ೚ҙͷΤοδΛ੾அ͢Δ͜ͱ͕Ͱ͖ΔɽNegative
linear Gate ͸ֶश͕ਐΉʹͭΕͯଛࣦΛ‫ݮ‬গͤ͞Δؔ਺Ͱ͋Γɼֶशͷং൫͸ Through Gateɼऴ൫͸ Cutoff Gateͱಉ͡ॲཧͱͳΔɽPositive linear Gate ͸ֶश͕ਐΉʹͭΕͯଛࣦΛ૿Ճͤ͞Δؔ਺Ͱ͋Γɼֶशͷং൫͸ Cutoff
Gateɼऴ൫͸ Through Gate ͱಉ͡ॲཧͱͳΔɽਤ 2: ෳ਺ͷ Generator ʹ͓͚Δ஌ࣝసҠάϥϑʢk = 3ʣ

3.3 ֶशํ๏
ఏҊख๏Ͱ͸ɼDiscriminator ͷֶश͓Αͼ GeneratorͷֶशΛަ‫܁ʹޓ‬Γฦ͢͜ͱͰ GANs ͷֶशΛߦ͏ɽ·ͨɼDiscriminator ͷֶश͸ MAD-GANs ͱಉ༷Ͱ͋Δɽ
Generator ͷֶश͸ɼ֤ Generator ͷੜ੒ը૾͕ Discriminator ʹΑΓ࣮ը૾ͱࣝผ͞ΕΔΑ͏ʹߦ͏ɽ·ͨɼఏҊख๏Ͱ͸஌ࣝసҠάϥϑʹԠͯ͡ɼ֤ Generator ͕‫ޓ‬͍ͷ஌ࣝΛసҠ͠ͳ͕Βֶश͢Δɽ֤ Generator ͷϊʔυؒͷΤοδ͸ Generator ಉ࢜ͷ஌ࣝͷ఻ୡΛද͠ɼi ൪໨ͷ Generator ͷΤοδ LD,i ͸ i ൪໨ͷ Generator ͕ੜ੒ͨ͠ੜ੒ը૾͕ Discriminator ʹ࣮ը૾ͱ൑அ͞Ε͔ͨͲ͏͔ͷ৘ใͷ఻ୡΛද͢ɽGenerator ͷϊʔυؒͷΤοδͰɼt ൪໨ͷ Generator ͷ஌ࣝΛ s ൪໨ͷ Generator ʹసҠ͢Δࡍͷଛࣦؔ਺Λ Lt,s ͱ͠ɼi ൪໨ͷ Generator ͕ੜ੒ͨ͠ੜ੒ը૾͕ Discriminator ʹ࣮ը૾ͱ൑அ͞Ε͔ͨͲ͏͔Λද͢ଛࣦؔ਺Λ LD,i ͱ͢Δɽ·ͨɼ֤Τοδʹ͓͚Δήʔτؔ਺͸ Gate(·) ͱ͢ΔɽGenerator ͷֶशͰ͸ɼ֤ Generator ʹೖྗ͞ΕΔજࡏม਺ z ∼ Pz ͸ಉҰͱ͢Δɽ࠷ऴతͳ֤ Generator ͷଛࣦؔ਺͸Τοδ͔ΒಘΒΕͨ Lt,s ͓Αͼ LD,i Λ૯࿨ͯ͠ࢉग़͢Δɽ࢖༻͢Δ
Generator ͷ਺Λ k ͱ͢ΔͱɼDiscriminator ʹରͯ͠ i൪໨ͷ Generator ͕ੜ੒ͨ͠ੜ੒ը૾Λೖྗͨ͠ࡍͷࣝผ(i)(i)݁Ռ͸ d(i) = d1 , . . . , dk+1 Ͱද͞ΕΔɽ·ͨɼਖ਼ղϥϕggg
ϧ͸ t = t1 , . . . , tk+1 Ͱද͞Εɼtgk+1 ͕ 1 Ͱͦͷଞ͕ 0ͷϕΫτϧͱͳΔɽ͜ͷͱ͖ɼଛࣦؔ਺ LD,i ͸ࣜ (1) ͱͳΔɽ
!
#LD,i = Gate −

k+1
"

(i)

tgj loge djද 1: IS ͓Αͼ FID ͷൺֱ݁Ռ

DCGANs

(m,n)



2

!

4.77 ± 0.05

21.17

4.77 ± 0.03

28.76

!

23.77

࣮ը૾

5.37 ± 0.09−

8.70 ± 0.14−(a) MAD-GANs(b) ఏҊख๏ਤ 3: ੜ੒ը૾ͷՄࢹԽ݁Ռ

(2)

2 
M
N
(m,n)
(m,n)
Qs
 α " "  ' Qt
' −'
'  
Lt,s = Gate ' (m,n) '  (m,n) '
M N m=1 n=1 '
'Qt
'
'Qs
'

25.79

ఏҊख๏

(1)

1 " 2
Fi,m,c,n
C c=1
4.57 ± 0.03

!

j=1

=

FID ↓

MAD-GANsఏҊख๏Ͱ͸ɼಛ௃ϚοϓΛసҠ͢Δ͜ͱʹΑΓɼ஌ࣝసҠΛߦ͏ɽಛ௃Ϛοϓͷνϟωϧ൪߸Λ c = 1, . . . , C ɼಛ௃Ϛοϓͷ૚ͷҐஔΛ m = 1, . . . , M ɼಛ௃ϚοϓͷϕΫτ
ϧ൪߸Λ n = 1 . . . , N ͱ͢Δͱɼi ൪໨ͷ Generator ͷಛ௃Ϛοϓ͸ Fi,m,c,n Ͱද͞ΕΔɽ͜ͷͱ͖ɼϋΠύʔύϥϝʔλΛ α ͱ͢Δͱɼଛࣦؔ਺ Lt,s ͸ࣜ (2) ͓Αͼࣜ (3)
ͱͳΔɽ
C
Qi

IS ↑

SN

4.3 ஌ࣝసҠάϥϑͷ࠷దԽ(3)

2

4. ධՁ࣮‫ݧ‬ैདྷख๏ͱͷൺֱʹΑΓɼఏҊख๏ͷ༗ޮੑΛௐࠪ͠ɼ࠷దͳ஌ࣝͷసҠํ๏Λ୳ࡧ͢Δɽ࣮‫ʹݧ‬͸ɼ10 ΫϥεͷҰൠ෺ମͷΧϥʔը૾Ͱߏ੒͞Ε͍ͯΔ CIFAR-10 σʔληοτΛ༻͍ɼֶश༻ͷαϯϓϧͰ͋Δ 50, 000 ຕΛ 64×64
ʹϦαΠζͯ͠࢖༻͢Δɽຊ࣮‫Ͱݧ‬͸ɼϕʔεͷ GANs ͷख๏ͱͯ͠ DCGANs Λ༻͍Δɽ·ͨɼ஌ࣝసҠάϥϑʹ͓͚Δήʔτؔ਺ͷ୳ࡧճ਺Λ 1, 500 ͱ͠ɼ֤ Generator ʹ͓͚Δੜ੒ը૾ͷ Inception ScoreʢISʣΛ࠷େԽ͢Δήʔτؔ਺ͷ૊Έ߹ΘͤΛ୳ࡧ͢Δɽֶश࣌ͷόοναΠζΛ256 ͱ͠ɼֶशճ਺Λ 50ɼ֤ Generator ʹೖྗ͢Δજࡏม਺ͷ࣍‫਺ݩ‬Λ 100ɼϋΠύʔύϥϝʔλ α Λ 1, 000 ͱ͢Δɽੜ੒ը૾ͷ࣭ΛධՁ͢Δࢦඪͱͯ͠ IS ͓Αͼ Fréchet
Inception DistanceʢFIDʣΛ༻͍Δɽ·ͨɼIS ͸ߴ͍஋ɼ
FID ͸௿͍஋ʹͳΔ΄Ͳੜ੒ը૾ͷ࣭͕ߴ͍͜ͱΛද͍ͯ͠ΔɽఏҊख๏ʹ͓͍ͯɼ஌ࣝసҠάϥϑΛ༻͍ͨήʔτؔ਺ͷ୳ࡧΛ 1, 500 ճߦ͍ɼ஌ࣝͷ఻ୡํ๏Λ࠷దԽͨ͠ɽ
࠷దԽର৅Ͱ͋Δ֤ Generator ʹ͓͚Δੜ੒ը૾ͷ IS ͕࠷େͷ஋ͱͳͬͨάϥϑΛਤ 4 ʹࣔ͢ɽ੺৭ͷΤοδ͕
Positive linear Gateɼփ৭ͷΤοδ͕ Through Gate Λද͍ͯ͠ΔɽG1 ͸ɼଞͷ Generator ͔Β஌ࣝΛసҠ͞Ε
ͳ͍ G2 ͱൺֱͯ͠ IS ͷ஋͕ߴ͘ɼ஌ࣝͷసҠ͸ੜ੒ը૾ͷ࣭ͷ޲্ʹ༗ޮͰ͋Δͱ͍͑Δɽ·ͨɼGenerator ؒͷ஌ࣝͷసҠ͸ֶशॳ‫ظ‬͸ෆཁͰ͋Γɼֶश͕ਐΉʹͭΕͯసҠ͢Δͱྑ͍͜ͱ΍ɼG3 ͸ G1 ͷόοϑΝͷΑ͏ͳ໾ׂΛՌͨ͢͜ͱͰɼG1 ͷֶशͷ҆ఆԽʹ‫د‬༩͍ͯ͠Δ͜
ͱ͕֬ೝͰ͖Δɽ

4.1 ఏҊख๏ͷ༗ޮੑͷௐࠪຊ࣮‫Ͱݧ‬͸ɼDCGANs ͓Αͼ MAD-GANsɼఏҊख๏ʹ͓͚Δੜ੒ը૾ͷ࣭Λൺֱ͢ΔɽDCGANs ͸ SN ͷಋೖͷ༗ແʹΑΔ̎ύλʔϯΛൺֱର৅ͱ͢ΔɽMAD-GANs
͓ΑͼఏҊख๏͸ SN Λಋೖ͠ɼGenerator ͷ਺Λ 3 ͱ͢
Δɽֶशճ਺Λ 200 Epoch ͱ͠ɼֶशऴྃ࣌ͷϞσϧύ
ϥϝʔλΛ༻͍ͯධՁΛߦ͏ɽ֤ख๏ͷ IS ͓Αͼ FID Λද 1 ʹࣔ͢ɽද 1 ΑΓɼఏҊख๏͸ IS ͕࠷΋ߴ͍஋ͱͳ
Γɼ࠷΋ੜ੒ը૾ͷ࣭͕ߴ͍͜ͱ͕֬ೝͰ͖ΔɽFID ͸ SNΛಋೖͨ͠ DCGANs ͕࠷΋௿͘ɼఏҊख๏΋ಉ༷ʹ௿͍
஋Ͱ͋ΔɽGAN ͸ֶश͕ਐΉͱɼIS ͱ FID ͕τϨʔυΦϑͷؔ܎ʹ͋Δ͜ͱ͕஌ΒΕ͍ͯΔɽຊ࣮‫͚͓ʹݧ‬ΔఏҊख๏͸ɼ֤ Generator ʹ͓͚Δੜ੒ը૾ͷ IS Λ࠷దԽର৅ͱ͠ɼ࠷େԽ͢Δήʔτؔ਺ͷ૊Έ߹ΘͤΛ୳ࡧͨ͠ɽ
ఏҊख๏͸࠷దԽର৅Λ FID ͱ͠ɼ࠷খԽ͢Δήʔτؔ਺ͷ૊Έ߹Θͤͷ୳ࡧʹΑΓɼFID ʹಛԽͨ͠ख๏΋࣮‫ݱ‬Ͱ͖Δɽ

4.2 ੜ੒ը૾ͷՄࢹԽ
MAD-GANs ͓ΑͼఏҊख๏ʹಉҰͷજࡏม਺Λೖྗ͠ɼੜ੒ͨ͠ը૾Λൺֱ͢Δɽੜ੒ը૾ͷՄࢹԽ݁ՌΛਤ 3 ʹࣔ͢ɽਤ 3 ͷ݁Ռ͔ΒɼఏҊख๏͸࣮ը૾ʹ͍ۙը૾͕ੜ
੒Ͱ͖͍ͯΔ͜ͱ͕֬ೝͰ͖Δɽਤ 4: ஌ࣝసҠάϥϑͷ࠷దԽ݁Ռ

5. ͓ΘΓʹ
ຊ‫Ͱڀݚ‬͸ෳ਺ͷ Generator ͓Αͼ̍ͭͷ Discriminator Λ༻͍ͯɼͦΕͧΕͷ Generator Ͱ஌ࣝΛసҠ͠ͳ͕Β‫ڞ‬ಉֶशΛߦ͏ख๏ΛఏҊͨ͠ɽධՁ࣮‫ʹݧ‬ΑΓఏҊख๏ͷ༗ޮੑΛ֬ೝͨ͠ɽࠓ‫ޙ‬͸ෳ਺ͷ Discriminator Λ༻͍Δ͜ͱͰɼΞϯαϯϒϧޮՌʹΑΓੜ੒ը૾ͷ͞ΒͳΔ࣭ͷ޲্Λ໨ࢦ͢ɽ

ࢀߟจ‫ݙ‬[1] A. Radford, et al., “Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks”, ICLR, 2016.
[2] A. Ghosh, et al., “Multi-Agent Diverse Generative Adversarial Networks”, CVPR, 2018.

‫੷ۀڀݚ‬[1] T. Tsukahara, et al., “Collaborative learning of generative adversarial networks”, VISAPP, 2021.
(ଞ 3 ݅)ಈը૾ʹ͓͚Δۭ࣌ؒ৘ใΛߟྀͨ͠ DNN ʹΑΔࢹ֮తઆ໌ʹؔ͢Δ‫ڀݚ‬ࢦಋ‫ڭ‬तɿ౻٢߂࿱

TR19010 ࡾ௡‫ݪ‬ক߂
1. ͸͡Ίʹ
ը૾ೝࣝ෼໺ʹ͓͍ͯɼਂ૚ֶशͷ൑அࠜ‫ڌ‬Λղੳ͢Δ
ख๏ͷҰͭʹࢹ֮తઆ໌͕͋Δɽࢹ֮తઆ໌Ͱ͸ɼਂ૚ֶश͕ೝࣝ͢Δࡍʹ஫ࢹͨ͠ྖҬΛώʔτϚοϓͰද‫ͨ͠ݱ‬
Attention map Λղੳ͢ΔɽҰํɼಈը૾ೝࣝͰ͸੩ࢭը૾ͱ͸ҟͳΓɼۭؒ৘ใ͚ͩͰͳ࣌ؒ͘৘ใ΋ߟྀ͢Δඞཁ͕
͋Δ͜ͱ͔Βɼ൑அࠜ‫ڌ‬ͷղੳ͕ࠔ೉ͱ͞Ε͖ͯͨɽຊ‫ڀݚ‬Ͱ͸ɼࢹ֮తઆ໌Λಈը૾ೝࣝʹ֦ு͠ɼಈը૾ʹ͓͚Δ৽
ͨͳࢹ֮తઆ໌ͷख๏Ͱ͋Δ Spatio-Temporal Attention
Branch Network (ST-ABN) ΛఏҊ͢ΔɽST-ABN ͸ɼਪ࿦࣌ͷۭؒ৘ใͱ࣌ؒ৘ใʹର͢Δॏཁ౓Λ֫ಘ͠ɼೝࣝ
ॲཧʹԠ༻͢Δ͜ͱͰೝࣝੑೳͷ޲্ͱࢹ֮తઆ໌ͷ֫ಘΛ࣮‫͢ݱ‬ΔɽධՁ࣮‫Ͱݧ‬͸ɼSomething-Something σʔληοτΛ༻͍࣮ͯ‫ݧ‬Λߦ͍ɼఏҊख๏ʹΑΓೝࣝੑೳͷ޲্ͱۭؒ৘ใͱɼ࣌ؒ৘ใΛಉ࣌ʹߟྀͨ͠ࢹ֮తઆ໌
͕ՄೳͱͳΔ͜ͱΛࣔ͢ɽ

2. ؔ࿈‫ڀݚ‬ैདྷख๏Ͱ͋Δ 3D CNN ‫֮ࢹͼٴ‬తઆ໌Λ༻͍ͨख๏
ʹ͍ͭͯड़΂Δɽ
3D Convolutional Networks 3D CNN ͷ୅දతͳ
ख๏Ͱ͋Δ 3D Convolutional Networks ʢC3Dʣ [1] ͸ɼۭؒํ޲ʹର͢Δ 2D ͷ৞ΈࠐΈॲཧΛ࣌ؒํ޲ʹ֦ு͠ɼ
3D ۭؒʹର͢Δ৞ΈࠐΈॲཧΛߦ͏͜ͱͰۭ࣌ؒͷಛ௃Λ֫ಘ͢Δɽ3D ۭؒʹରͯ͠৞ΈࠐΈॲཧΛߦ͏͜ͱͰɼ
ैདྷͷۭؒํ޲ʹର͢Δ৞ΈࠐΈॲཧΛߦ͏ 2D CNN Ͱ͸ࠔ೉Ͱ͋ͬͨ࣌‫ྻܥ‬৘ใΛଊ͑Δ͜ͱ͕Ͱ͖Δɽ
Attention Branch Network ࢹ֮తઆ໌Λ༻͍ͨख๏ʹ Attention Branch Network (ABN) [2] ͕͋ΔɽABN͸ɼAttention map Λ Attention ‫ʹߏػ‬ΑΓಛ௃ϚοϓʹॏΈ෇͚͢Δ͜ͱͰೝࣝੑೳͱࢹ֮తͳઆ໌ੑͷ޲্Λ࣮‫͍ͯ͠ݱ‬Δɽ

3. ఏҊख๏
ຊ‫Ͱڀݚ‬͸ɼॏཁͳۭؒ৘ใͱ࣌ؒ৘ใΛಉ࣌ʹߟྀͨ͠ࢹ֮తઆ໌͕Մೳͳ ST-ABN ΛఏҊ͢Δɽ

3.1 ST-ABN ͷߏ଄
ST-ABN ͸ɼਤ 1 ʹࣔ͢Α͏ʹ Feature extractorɼSpatioTemporal (ST) Attention branchɼPerception branch ͷ
3 ͭͷϞδϡʔϧͰߏ੒͢ΔɽFeature extractor ͸ɼෳ਺ͷ৞ΈࠐΈ૚Ͱߏ੒͞Ε͓ͯΓɼೖྗ͔Βಛ௃ϚοϓΛ֫ಘ͢Δɽਤ 2 : ST Attention branch ͷߏ଄

3.2 Spatio-Temporal Attention branch
ST Attention branch Ͱ͸ɼਤ 2 ʹࣔ͢Α͏ʹ Feature
extractor ͔Βग़ྗ͞Εͨಛ௃ϚοϓΛ༻͍ͯɼۭؒ৘ใ
ͱ֤ϑϨʔϜʹର͢Δॏཁ౓Λ֫ಘ͠ɼGlobal Average
Pooling (GAP) Λհͯ͠ΫϥεࣝผΛߦ͏ɽFeature extractor ͔Βग़ྗ͞Εͨಛ௃Ϛοϓ͸ɼෳ਺ͷ৞ΈࠐΈ૚Λ‫ͯܦ‬ɼΫϥε਺෼ͷνϟωϧΛ࣋ͭ 1 × 1 ͷ৞ΈࠐΈ૚ʹΑΓΫϥε਺෼ͷಛ௃ϚοϓΛ֫ಘ͢Δɽ
Spatial attention ͸ɼ͜ͷΫϥε਺෼ͷಛ௃ϚοϓΛ༻͍ͯੜ੒͢ΔɽΫϥε਺෼ͷಛ௃Ϛοϓ͸ɼ1 × 1 ͷ৞ΈࠐΈ૚ʹΑΓ 1 ຕͷಛ௃Ϛοϓʹू໿͢Δɽͦͷ‫ޙ‬ɼϑϨʔϜ͝ͱʹ Sigmoid ؔ਺Ͱ 0 ͔Β 1 ͷൣғʹਖ਼‫ن‬Խͨ͠ Attention map Λ֫ಘ͢Δɽ
Temporal attention ΋ Spatial attention ͱಉ༷ʹΫϥε਺෼ͷಛ௃ϚοϓΛ༻͍ͯੜ੒͢Δɽ͸͡ΊʹɼΫϥε਺෼ͷಛ௃ϚοϓΛ 1 × 1 ͷ৞ΈࠐΈ૚ʹΑΓνϟωϧํ޲ʹରͯ࣍͠‫ݩ‬Λѹॖ͢Δɽͦͷ‫ޙ‬ɼϑϨʔϜ਺෼ͷνϟωϧ਺Λ࣋ͭ৞ΈࠐΈ૚ͱ GAP Λհͯ͠ɼ֤ಛ௃Ϛοϓͷۭؒํ޲ʹର͢Δฏ‫ۉ‬஋Λ‫ٻ‬ΊΔɽ࠷‫ʹޙ‬ɼશ݁߹૚ɼ
ReLU ‫ ͼٴ‬Sigmoid ؔ਺Λհͯ͠ɼ֤ϑϨʔϜͷॏཁ౓Λ֫ಘ͢Δɽ

3.3 Attention ‫ߏػ‬
Spatial attention Ms (xi ) ͸ɼࣜ (2) ΑΓಛ௃Ϛοϓ f (xi )ʹॏΈ෇͚͠ɼॏΈ෇͚લͷಛ௃ϚοϓΛՃࢉ͢Δɽ͜ΕʹΑΓɼಛ௃ϚοϓͷফࣦΛ཈੍͠ɼAttention map Λޮ཰తʹೝࣝʹ൓өͤ͞Δ͜ͱ͕Ͱ͖Δɽ
fs! (xi ) = (1 + Ms (xi ) · f (xi ))(2)

Temporal attention Mt (xi ) ͸ɼࣜ (3) ΑΓಛ௃Ϛοϓʹ৐ࢉ͢Δ͜ͱͰॏΈ෇͚Λߦ͏ɽ
ft! (xi ) = Mt (xi ) · f (xi )ਤ 1 : ST-ABN ͷωοτϫʔΫߏ଄

ST Attention branch ͸ɼۭؒ৘ใʹର͢Δॏཁ౓Λࣔ͢ Spatial attention ͱ࣌ؒ৘ใʹର͢Δॏཁ౓Λࣔ͢
Temporal attention Λ֫ಘ͢ΔɽPerception branch ͸ɼ
Attention ‫ʹߏػ‬ΑΓ Spatial attention ͱ Temporal attention ΛॏΈ෇͚ͨ͠ಛ௃ϚοϓΛೖྗ͠ɼ֤Ϋϥεͷ֬཰Λग़ྗ͢ΔɽST-ABN ͸ɼࣜ (1) ͷΑ͏ʹ ST Attention
branch ͷֶश‫ ࠩޡ‬Latt ͱ Perception branch ͷֶश‫ࠩޡ‬
Lper Λ༻ֶ͍ͯश͢ΔɽΫϥεࣝผ‫ࠩޡ‬͸ɼSoftmax ؔ਺ͱΫϩεΤϯτϩϐʔ‫ࠩޡ‬Λ༻͍ͯࢉग़͢Δɽ
L(xi ) = Latt (xi ) + Lper (xi )

(1)

(3)

Spatial attention ͱ Temporal attention ͰͦΕͧΕॏΈ෇͚ͨ͠ಛ௃Ϛοϓ͸ɼࣜ (4) ΑΓνϟωϧํ޲ʹ݁߹
͢Δɽͦͷ‫ޙ‬ɼ݁߹ͨ͠ಛ௃ϚοϓΛ৞ΈࠐΈ૚ʹೖྗͯ͠౷߹͢Δɽ
!
"f ! (xi ) = conv fs! (xi ) " ft! (xi )(4)͜ͷ౷߹ͨ͠ಛ௃ϚοϓΛ Perception branch ʹೖྗ͠ɼ࠷ऴతͳೝࣝ݁ՌΛग़ྗ͢Δɽ͜ΕʹΑΓɼॏཁͳۭ࣌ؒ
ಛ௃ʹண໨ֶͨ͠श͕ՄೳͱͳΔɽ

4. ධՁ࣮‫ݧ‬ධՁ࣮‫Ͱݧ‬͸ɼSomething-Something σʔληοτ [3]ͷ V1 ͱ V2 Λ༻͍ͯϕʔεϥΠϯͱఏҊख๏ͱͷೝࣝਫ਼౓Λൺֱ͢ΔɽSomething-Something σʔληοτ͸ɼେ‫ن‬໛ͳಈ࡞ೝࣝ༻ͷσʔληοτͰ͋Γɼਓ͕೔༻඼Λѻ
͏ 174 छྨͷ‫ج‬ຊతͳಈ࡞Λೝࣝ͢ΔɽఏҊख๏͸ɼϕʔεϥΠϯͰ͋Δ 3D ResNet-50 Λϕʔεʹߏங͢Δɽೖྗਤ 3 : Spatial attention ͱ Temporal attention ͷՄࢹԽ݁Ռ
ද 1 : ϕʔεϥΠϯͱఏҊख๏ͱͷೝࣝਫ਼౓ͷൺֱ݁Ռ [%]Model

Frames3D ResNet-50 (ϕʔεϥΠϯ)3D ResNet-50 + ఏҊख๏
3D ResNet-50 (ϕʔεϥΠϯ)3D ResNet-50 + ఏҊख๏

32
32
32+32
32+32

Something-Something V1
Top-1
Top-5
45.2
74.5
45.9
75.4
52.9
81.5
53.3
82.0

͢ΔϑϨʔϜ਺͸ɼ32 ϑϨʔϜͱ 64 ϑϨʔϜΛೖྗͨ͠৔߹Ͱೝࣝਫ਼౓Λൺֱ͢Δɽ64 ϑϨʔϜΛೖྗ͢Δ৔߹͸ɼ32 ϑϨʔϜͷಈըΛ 2 ͭೖྗ͠ɼ2 ͭͷಈըʹର͢Δ
είΞΛฏ‫͢ۉ‬Δ͜ͱͰ࠷ऴతͳೝࣝਫ਼౓Λࢉग़͢Δɽ

4.1 ϕʔεϥΠϯͱͷೝࣝਫ਼౓ͷൺֱ
ද 1 ʹϕʔεϥΠϯͱఏҊख๏ͱͷೝࣝਫ਼౓ͷൺֱ݁ՌΛࣔ͢ɽද 1 ʹࣔ͢Α͏ʹఏҊख๏͕ϕʔεϥΠϯΑΓ΋
ߴ͍ೝࣝਫ਼౓Λ֫ಘͨ͠ɽ͞ΒʹϑϨʔϜ਺Λ૿΍ͨ͠৔߹ʹ͓͍ͯ΋ఏҊख๏͕ϕʔεϥΠϯΑΓ΋ߴ͍ೝࣝਫ਼౓
Λ֫ಘͨ͠ɽ

4.2 Attention ͷఆੑతͳධՁ
ਤ 3 ʹ Something-Something σʔληοτ V2 ʹ͓͚
Δ Spatial attention ͱ Temporal attention ͷՄࢹԽ݁Ռ
Λࣔ͢ɽਤ 3 ͷ্͔ΒԼʹ޲͔ͬͯೖྗϑϨʔϜɼTemporalattentionɼSpatial attentionɼೖྗಈըͷΫϥεΛ͍ࣔͯ͠ΔɽTemporal attention Ͱ͸ɼ֤ϑϨʔϜʹରͯ͠ग़ྗ
͞Εͨॏཁ౓ΛώʔτϚοϓͷ৭ʹม‫͢׵‬Δ͜ͱͰՄࢹԽ
͓ͯ͠Γɼॏཁ౓͕ߴ͍΄Ͳ੺͘ɼॏཁ౓͕௿͍΄Ͳ੨͘ͳΔɽਤ 3 ʹࣔ͢Α͏ʹɼಈ͖Λ‫ؚ‬ΉϑϨʔϜͷॏཁ౓͕
ߴ͘ͳΔ͜ͱ͕֬ೝͰ͖Δɽ
Spatial attention Ͱ͸ɼ֤ϑϨʔϜʹରͯ͠ग़ྗ͞Εͨ
Attention map ΛώʔτϚοϓͱͯ͠ՄࢹԽ͢ΔɽSpatialattention ͸ɼಈ࡞ಛ༗ͷಛ௃తͳۭؒྖҬΛଊ͑ͭͭɼಈ
తͳϑϨʔϜʹରͯ͠‫͘ڧ‬஫ࢹ͍ͯ͠Δ͜ͱ͕֬ೝͰ͖Δɽ͜ΕΒͷ݁Ռ͔ΒɼఏҊख๏ʹΑΓۭؒ৘ใͱ࣌ؒ৘ใΛಉ࣌ʹߟྀͨ͠ࢹ֮తઆ໌͕ՄೳͰ͋Δɽ

4.3 Attention ͷ༗ޮੑͷධՁ
Spatial attention ͱ Temporal attention ͷ༗ޮੑΛఆྔతʹධՁ͢ΔɽධՁํ๏ͱͯ͠ Spatial attention ͱ Temporal attention Λ൓సͤͯ͞ਪ࿦Λߦ͏ɽͦͯ͠ɼ൓స͢
Δ৔߹ͱ൓స͠ͳ͍৔߹Ͱೝࣝਫ਼౓͕ͲΕ͚ͩ௿Լ͢Δ͔
Λௐࠪ͠ɼೝࣝʹ༗ӹͳۭؒ৘ใͱ࣌ؒ৘ใʹର͢Δॏཁ౓͕ಘΒΕ͍ͯΔ͜ͱΛ֬ೝ͢Δɽ
ද 2 ʹ Something-Something σʔληοτʹ͓͚Δ
Spatial attention ͱ Temporal attention ͷ൓సʹΑΔೝ

Something-Something V2
Top-1
Top-5
55.6
81.6
56.6
82.0
63.8
89.2
64.1
89.6

ද 2 : Attention ͷ൓సʹΑΔೝࣝਫ਼౓ͷൺֱ݁Ռ [%]Attention
SpatialTemporal!
!

!
!

Something. V1
Top-1
Top-5
45.9
75.4
36.2
66.8
28.3
57.4
22.7
50.0

Something. V2
Top-1
Top-5
56.6
82.0
19.1
39.7
45.5
75.6
12.4
29.0

ࣝਫ਼౓ͷൺֱ݁ՌΛࣔ͢ɽද 2 ʹࣔ͢Α͏ʹɼSpatial attention ͷΈͷ൓సͱ Temporal attention ͷΈͷ൓సʹΑΓೝࣝਫ਼౓͕௿Լ͢Δ͜ͱ͕֬ೝͰ͖Δɽ͞ΒʹɼSpatialattention ͱ Temporal attention Λ൓సͤ͞Δ͜ͱͰେ෯
ʹೝࣝਫ਼౓͕௿Լ͢Δ͜ͱ͔Βೝࣝʹ༗ӹͳॏཁ౓͕ಘΒΕ͍ͯΔ͜ͱ͕֬ೝͰ͖Δɽ

5. ͓ΘΓʹ
ຊ‫Ͱڀݚ‬͸ɼST-ABN ΛఏҊ͠ɼಈը૾ೝࣝʹ͓͚Δۭؒ৘ใͱ࣌ؒ৘ใΛಉ࣌ʹߟྀͨ͠ࢹ֮తઆ໌Λ࣮‫͠ݱ‬
ͨɽST-ABN ͸ɼ3D CNN ͕ϕʔεͰ͋ΔϞσϧͷਪ࿦࣌ʹ͓͚Δۭؒ৘ใͱ࣌ؒ৘ใʹର͢Δॏཁ౓Λ֫ಘ͠ɼ
Attention ‫ʹߏػ‬Ԡ༻͢Δ͜ͱͰࢹ֮తͳઆ໌ੑͱೝࣝੑ
ೳͷ޲্͕ՄೳͱͳΔɽࠓ‫ޙ‬͸ɼఏҊख๏Λ෼ྨҎ֎ͷଞͷಈը૾ೝࣝλεΫ΁Ԡ༻͢Δ͜ͱΛ‫ݕ‬౼͢Δɽ

ࢀߟจ‫ݙ‬[1] D. Tran, et al., “Learning Spatiotemporal Features with
3D Convolutional Networks”, ICCV, 2015.
[2] H. Fukui, et al., “Attention Branch Network: Learningof Attention Mechanism for Visual Explanation”, CVPR,
2019.
[3] R. Goyal, et al., “The“Something Something” Video
Database for Learning and Evaluating Visual Common
Sense”, ICCV, 2017.

‫੷ۀڀݚ‬[1] ࡾ௡‫ݪ‬ক߂ ౳, “Attention map Λհͨ͠ਓͷ஌‫ݟ‬ͷ૊ΈࠐΈ”, ୈ 22 ճ ը૾ͷೝࣝɾཧղγϯϙδ΢Ϝ, 2019.
[2] M. Mitsuhara, et al., “Embedding Human Knowledge
into Deep Neural Network via Attention Map”, VISAPP,
2021.