ಈը૾ʹ͓͚Δۭ࣌ؒ৘ใΛߟྀͨ͠ DNN ʹΑΔࢹ֮తઆ໌ʹؔ͢Δ‫ڀݚ‬ࢦಋ‫ڭ‬तɿ౻٢߂࿱

TR19010 ࡾ௡‫ݪ‬ক߂
1. ͸͡Ίʹը૾ೝࣝ෼໺ʹ͓͍ͯɼਂ૚ֶशͷ൑அࠜ‫ڌ‬Λղੳ͢Δख๏ͷҰͭʹࢹ֮తઆ໌͕͋Δɽࢹ֮తઆ໌Ͱ͸ɼਂ૚ֶश͕ೝࣝ͢Δࡍʹ஫ࢹͨ͠ྖҬΛώʔτϚοϓͰද‫ͨ͠ݱ‬
Attention map Λղੳ͢ΔɽҰํɼಈը૾ೝࣝͰ͸੩ࢭը૾ͱ͸ҟͳΓɼۭؒ৘ใ͚ͩͰͳ࣌ؒ͘৘ใ΋ߟྀ͢Δඞཁ͕͋Δ͜ͱ͔Βɼ൑அࠜ‫ڌ‬ͷղੳ͕ࠔ೉ͱ͞Ε͖ͯͨɽຊ‫ڀݚ‬Ͱ͸ɼࢹ֮తઆ໌Λಈը૾ೝࣝʹ֦ு͠ɼಈը૾ʹ͓͚Δ৽ͨͳࢹ֮తઆ໌ͷख๏Ͱ͋Δ Spatio-Temporal Attention
Branch Network (ST-ABN) ΛఏҊ͢ΔɽST-ABN ͸ɼਪ࿦࣌ͷۭؒ৘ใͱ࣌ؒ৘ใʹର͢Δॏཁ౓Λ֫ಘ͠ɼೝࣝ
ॲཧʹԠ༻͢Δ͜ͱͰೝࣝੑೳͷ޲্ͱࢹ֮తઆ໌ͷ֫ಘΛ࣮‫͢ݱ‬ΔɽධՁ࣮‫Ͱݧ‬͸ɼSomething-Something σʔληοτΛ༻͍࣮ͯ‫ݧ‬Λߦ͍ɼఏҊख๏ʹΑΓೝࣝੑೳͷ޲্ͱۭؒ৘ใͱɼ࣌ؒ৘ใΛಉ࣌ʹߟྀͨ͠ࢹ֮తઆ໌͕ՄೳͱͳΔ͜ͱΛࣔ͢ɽ

2. ؔ࿈‫ڀݚ‬ैདྷख๏Ͱ͋Δ 3D CNN ‫֮ࢹͼٴ‬తઆ໌Λ༻͍ͨख๏ʹ͍ͭͯड़΂Δɽ
3D Convolutional Networks 3D CNN ͷ୅දతͳ
ख๏Ͱ͋Δ 3D Convolutional Networks ʢC3Dʣ [1] ͸ɼۭؒํ޲ʹର͢Δ 2D ͷ৞ΈࠐΈॲཧΛ࣌ؒํ޲ʹ֦ு͠ɼ
3D ۭؒʹର͢Δ৞ΈࠐΈॲཧΛߦ͏͜ͱͰۭ࣌ؒͷಛ௃Λ֫ಘ͢Δɽ3D ۭؒʹରͯ͠৞ΈࠐΈॲཧΛߦ͏͜ͱͰɼैདྷͷۭؒํ޲ʹର͢Δ৞ΈࠐΈॲཧΛߦ͏ 2D CNN Ͱ͸ࠔ೉Ͱ͋ͬͨ࣌‫ྻܥ‬৘ใΛଊ͑Δ͜ͱ͕Ͱ͖Δɽ
Attention Branch Network ࢹ֮తઆ໌Λ༻͍ͨख๏ʹ Attention Branch Network (ABN) [2] ͕͋ΔɽABN͸ɼAttention map Λ Attention ‫ʹߏػ‬ΑΓಛ௃ϚοϓʹॏΈ෇͚͢Δ͜ͱͰೝࣝੑೳͱࢹ֮తͳઆ໌ੑͷ޲্Λ࣮‫͍ͯ͠ݱ‬Δɽ

3. ఏҊख๏
ຊ‫Ͱڀݚ‬͸ɼॏཁͳۭؒ৘ใͱ࣌ؒ৘ใΛಉ࣌ʹߟྀͨ͠ࢹ֮తઆ໌͕Մೳͳ ST-ABN ΛఏҊ͢Δɽ

3.1 ST-ABN ͷߏ଄
ST-ABN ͸ɼਤ 1 ʹࣔ͢Α͏ʹ Feature extractorɼSpatioTemporal (ST) Attention branchɼPerception branch ͷ3 ͭͷϞδϡʔϧͰߏ੒͢ΔɽFeature extractor ͸ɼෳ਺ͷ৞ΈࠐΈ૚Ͱߏ੒͞Ε͓ͯΓɼೖྗ͔Βಛ௃ϚοϓΛ֫ಘ͢Δɽਤ 2 : ST Attention branch ͷߏ଄

3.2 Spatio-Temporal Attention branch
ST Attention branch Ͱ͸ɼਤ 2 ʹࣔ͢Α͏ʹ Featureextractor ͔Βग़ྗ͞Εͨಛ௃ϚοϓΛ༻͍ͯɼۭؒ৘ใͱ֤ϑϨʔϜʹର͢Δॏཁ౓Λ֫ಘ͠ɼGlobal Average
Pooling (GAP) Λհͯ͠ΫϥεࣝผΛߦ͏ɽFeature extractor ͔Βग़ྗ͞Εͨಛ௃Ϛοϓ͸ɼෳ਺ͷ৞ΈࠐΈ૚Λ‫ͯܦ‬ɼΫϥε਺෼ͷνϟωϧΛ࣋ͭ 1 × 1 ͷ৞ΈࠐΈ૚ʹΑΓΫϥε਺෼ͷಛ௃ϚοϓΛ֫ಘ͢Δɽ
Spatial attention ͸ɼ͜ͷΫϥε਺෼ͷಛ௃ϚοϓΛ༻͍ͯੜ੒͢ΔɽΫϥε਺෼ͷಛ௃Ϛοϓ͸ɼ1 × 1 ͷ৞ΈࠐΈ૚ʹΑΓ 1 ຕͷಛ௃Ϛοϓʹू໿͢Δɽͦͷ‫ޙ‬ɼϑϨʔϜ͝ͱʹ Sigmoid ؔ਺Ͱ 0 ͔Β 1 ͷൣғʹਖ਼‫ن‬Խͨ͠ Attention map Λ֫ಘ͢Δɽ
Temporal attention ΋ Spatial attention ͱಉ༷ʹΫϥε਺෼ͷಛ௃ϚοϓΛ༻͍ͯੜ੒͢Δɽ͸͡ΊʹɼΫϥε਺෼ͷಛ௃ϚοϓΛ 1 × 1 ͷ৞ΈࠐΈ૚ʹΑΓνϟωϧํ޲ʹରͯ࣍͠‫ݩ‬Λѹॖ͢Δɽͦͷ‫ޙ‬ɼϑϨʔϜ਺෼ͷνϟωϧ਺Λ࣋ͭ৞ΈࠐΈ૚ͱ GAP Λհͯ͠ɼ֤ಛ௃Ϛοϓͷۭؒํ޲ʹର͢Δฏ‫ۉ‬஋Λ‫ٻ‬ΊΔɽ࠷‫ʹޙ‬ɼશ݁߹૚ɼ
ReLU ‫ ͼٴ‬Sigmoid ؔ਺Λհͯ͠ɼ֤ϑϨʔϜͷॏཁ౓Λ֫ಘ͢Δɽ

3.3 Attention ‫ߏػ‬
Spatial attention Ms (xi ) ͸ɼࣜ (2) ΑΓಛ௃Ϛοϓ f (xi )ʹॏΈ෇͚͠ɼॏΈ෇͚લͷಛ௃ϚοϓΛՃࢉ͢Δɽ͜ΕʹΑΓɼಛ௃ϚοϓͷফࣦΛ཈੍͠ɼAttention map Λޮ཰తʹೝࣝʹ൓өͤ͞Δ͜ͱ͕Ͱ͖Δɽfs′ (xi ) = (1 + Ms (xi ) · f (xi ))(2)

Temporal attention Mt (xi ) ͸ɼࣜ (3) ΑΓಛ௃Ϛοϓʹ৐ࢉ͢Δ͜ͱͰॏΈ෇͚Λߦ͏ɽft′ (xi ) = Mt (xi ) · f (xi )ਤ 1 : ST-ABN ͷωοτϫʔΫߏ଄

ST Attention branch ͸ɼۭؒ৘ใʹର͢Δॏཁ౓Λࣔ͢ Spatial attention ͱ࣌ؒ৘ใʹର͢Δॏཁ౓Λࣔ͢
Temporal attention Λ֫ಘ͢ΔɽPerception branch ͸ɼ
Attention ‫ʹߏػ‬ΑΓ Spatial attention ͱ Temporal attention ΛॏΈ෇͚ͨ͠ಛ௃ϚοϓΛೖྗ͠ɼ֤Ϋϥεͷ֬཰Λग़ྗ͢ΔɽST-ABN ͸ɼࣜ (1) ͷΑ͏ʹ ST Attentionbranch ͷֶश‫ ࠩޡ‬Latt ͱ Perception branch ͷֶश‫ࠩޡ‬
Lper Λ༻ֶ͍ͯश͢ΔɽΫϥεࣝผ‫ࠩޡ‬͸ɼSoftmax ؔ਺ͱΫϩεΤϯτϩϐʔ‫ࠩޡ‬Λ༻͍ͯࢉग़͢Δɽ
L(xi ) = Latt (xi ) + Lper (xi )

(1)

(3)

Spatial attention ͱ Temporal attention ͰͦΕͧΕॏΈ෇͚ͨ͠ಛ௃Ϛοϓ͸ɼࣜ (4) ΑΓνϟωϧํ޲ʹ݁߹
͢Δɽͦͷ‫ޙ‬ɼ݁߹ͨ͠ಛ௃ϚοϓΛ৞ΈࠐΈ૚ʹೖྗͯ͠౷߹͢Δɽ
!
"f ′ (xi ) = conv fs′ (xi ) ⊙ ft′ (xi )(4)͜ͷ౷߹ͨ͠ಛ௃ϚοϓΛ Perception branch ʹೖྗ͠ɼ࠷ऴతͳೝࣝ݁ՌΛग़ྗ͢Δɽ͜ΕʹΑΓɼॏཁͳۭ࣌ؒಛ௃ʹண໨ֶͨ͠श͕ՄೳͱͳΔɽ

4. ධՁ࣮‫ݧ‬ධՁ࣮‫Ͱݧ‬͸ɼSomething-Something σʔληοτ [3]ͷ V1 ͱ V2 Λ༻͍ͯϕʔεϥΠϯͱఏҊख๏ͱͷೝࣝਫ਼౓Λൺֱ͢ΔɽSomething-Something σʔληοτ͸ɼେ‫ن‬໛ͳಈ࡞ೝࣝ༻ͷσʔληοτͰ͋Γɼਓ͕೔༻඼Λѻ
͏ 174 छྨͷ‫ج‬ຊతͳಈ࡞Λೝࣝ͢ΔɽఏҊख๏͸ɼϕʔεϥΠϯͰ͋Δ 3D ResNet-50 Λϕʔεʹߏங͢Δɽೖྗਤ 3 : Spatial attention ͱ Temporal attention ͷՄࢹԽ݁Ռ
ද 1 : ϕʔεϥΠϯͱఏҊख๏ͱͷೝࣝਫ਼౓ͷൺֱ݁Ռ [%]ModelFrames

3D ResNet-50 (ϕʔεϥΠϯ)3D ResNet-50 + ఏҊख๏
3D ResNet-50 (ϕʔεϥΠϯ)3D ResNet-50 + ఏҊख๏323232+32
32+32

Something-Something V1Top-1Top-5
45.2
74.5
45.9
75.4
52.9
81.5
53.3
82.0

͢ΔϑϨʔϜ਺͸ɼ32 ϑϨʔϜͱ 64 ϑϨʔϜΛೖྗͨ͠৔߹Ͱೝࣝਫ਼౓Λൺֱ͢Δɽ64 ϑϨʔϜΛೖྗ͢Δ৔߹͸ɼ32 ϑϨʔϜͷಈըΛ 2 ͭೖྗ͠ɼ2 ͭͷಈըʹର͢Δ
είΞΛฏ‫͢ۉ‬Δ͜ͱͰ࠷ऴతͳೝࣝਫ਼౓Λࢉग़͢Δɽ

4.1 ϕʔεϥΠϯͱͷೝࣝਫ਼౓ͷൺֱ
ද 1 ʹϕʔεϥΠϯͱఏҊख๏ͱͷೝࣝਫ਼౓ͷൺֱ݁ՌΛࣔ͢ɽද 1 ʹࣔ͢Α͏ʹఏҊख๏͕ϕʔεϥΠϯΑΓ΋
ߴ͍ೝࣝਫ਼౓Λ֫ಘͨ͠ɽ͞ΒʹϑϨʔϜ਺Λ૿΍ͨ͠৔߹ʹ͓͍ͯ΋ఏҊख๏͕ϕʔεϥΠϯΑΓ΋ߴ͍ೝࣝਫ਼౓
Λ֫ಘͨ͠ɽ

4.2 Attention ͷఆੑతͳධՁ
ਤ 3 ʹ Something-Something σʔληοτ V2 ʹ͓͚
Δ Spatial attention ͱ Temporal attention ͷՄࢹԽ݁ՌΛࣔ͢ɽਤ 3 ͷ্͔ΒԼʹ޲͔ͬͯೖྗϑϨʔϜɼTemporalattentionɼSpatial attentionɼೖྗಈըͷΫϥεΛ͍ࣔͯ͠ΔɽTemporal attention Ͱ͸ɼ֤ϑϨʔϜʹରͯ͠ग़ྗ͞Εͨॏཁ౓ΛώʔτϚοϓͷ৭ʹม‫͢׵‬Δ͜ͱͰՄࢹԽ͓ͯ͠Γɼॏཁ౓͕ߴ͍΄Ͳ੺͘ɼॏཁ౓͕௿͍΄Ͳ੨͘ͳΔɽਤ 3 ʹࣔ͢Α͏ʹɼಈ͖Λ‫ؚ‬ΉϑϨʔϜͷॏཁ౓͕
ߴ͘ͳΔ͜ͱ͕֬ೝͰ͖Δɽ
Spatial attention Ͱ͸ɼ֤ϑϨʔϜʹରͯ͠ग़ྗ͞Εͨ
Attention map ΛώʔτϚοϓͱͯ͠ՄࢹԽ͢ΔɽSpatialattention ͸ɼಈ࡞ಛ༗ͷಛ௃తͳۭؒྖҬΛଊ͑ͭͭɼಈతͳϑϨʔϜʹରͯ͠‫͘ڧ‬஫ࢹ͍ͯ͠Δ͜ͱ͕֬ೝͰ͖Δɽ͜ΕΒͷ݁Ռ͔ΒɼఏҊख๏ʹΑΓۭؒ৘ใͱ࣌ؒ৘ใΛಉ࣌ʹߟྀͨ͠ࢹ֮తઆ໌͕ՄೳͰ͋Δɽ

4.3 Attention ͷ༗ޮੑͷධՁ
Spatial attention ͱ Temporal attention ͷ༗ޮੑΛఆྔతʹධՁ͢ΔɽධՁํ๏ͱͯ͠ Spatial attention ͱ Temporal attention Λ൓సͤͯ͞ਪ࿦Λߦ͏ɽͦͯ͠ɼ൓స͢Δ৔߹ͱ൓స͠ͳ͍৔߹Ͱೝࣝਫ਼౓͕ͲΕ͚ͩ௿Լ͢Δ͔Λௐࠪ͠ɼೝࣝʹ༗ӹͳۭؒ৘ใͱ࣌ؒ৘ใʹର͢Δॏཁ౓͕ಘΒΕ͍ͯΔ͜ͱΛ֬ೝ͢Δɽ
ද 2 ʹ Something-Something σʔληοτʹ͓͚Δ
Spatial attention ͱ Temporal attention ͷ൓సʹΑΔೝ

Something-Something V2
Top-1
Top-5
55.6
81.6
56.6
82.0
63.8
89.2
64.1
89.6

ද 2 : Attention ͷ൓సʹΑΔೝࣝਫ਼౓ͷൺֱ݁Ռ [%]Attention
SpatialTemporal!
!

!
!

Something. V1
Top-1
Top-5
45.9
75.4
36.2
66.8
28.3
57.4
22.7
50.0

Something. V2
Top-1
Top-5
56.6
82.0
19.1
39.7
45.5
75.6
12.4
29.0

ࣝਫ਼౓ͷൺֱ݁ՌΛࣔ͢ɽද 2 ʹࣔ͢Α͏ʹɼSpatial attention ͷΈͷ൓సͱ Temporal attention ͷΈͷ൓సʹΑΓೝࣝਫ਼౓͕௿Լ͢Δ͜ͱ͕֬ೝͰ͖Δɽ͞ΒʹɼSpatialattention ͱ Temporal attention Λ൓సͤ͞Δ͜ͱͰେ෯ʹೝࣝਫ਼౓͕௿Լ͢Δ͜ͱ͔Βೝࣝʹ༗ӹͳॏཁ౓͕ಘΒΕ͍ͯΔ͜ͱ͕֬ೝͰ͖Δɽ

5. ͓ΘΓʹຊ‫Ͱڀݚ‬͸ɼST-ABN ΛఏҊ͠ɼಈը૾ೝࣝʹ͓͚Δۭؒ৘ใͱ࣌ؒ৘ใΛಉ࣌ʹߟྀͨ͠ࢹ֮తઆ໌Λ࣮‫͠ݱ‬
ͨɽST-ABN ͸ɼ3D CNN ͕ϕʔεͰ͋ΔϞσϧͷਪ࿦࣌ʹ͓͚Δۭؒ৘ใͱ࣌ؒ৘ใʹର͢Δॏཁ౓Λ֫ಘ͠ɼ
Attention ‫ʹߏػ‬Ԡ༻͢Δ͜ͱͰࢹ֮తͳઆ໌ੑͱೝࣝੑೳͷ޲্͕ՄೳͱͳΔɽࠓ‫ޙ‬͸ɼఏҊख๏Λ෼ྨҎ֎ͷଞͷಈը૾ೝࣝλεΫ΁Ԡ༻͢Δ͜ͱΛ‫ݕ‬౼͢Δɽࢀߟจ‫ݙ‬[1] D. Tran, et al., “Learning Spatiotemporal Features with
3D Convolutional Networks”, ICCV, 2015.
[2] H. Fukui, et al., “Attention Branch Network: Learningof Attention Mechanism for Visual Explanation”, CVPR,
2019.
[3] R. Goyal, et al., “The“Something Something” VideoDatabase for Learning and Evaluating Visual CommonSense”, ICCV, 2017.

‫੷ۀڀݚ‬[1] ࡾ௡‫ݪ‬ক߂ ౳, “Attention map Λհͨ͠ਓͷ஌‫ݟ‬ͷ૊ΈࠐΈ”, ୈ 22 ճ ը૾ͷೝࣝɾཧղγϯϙδ΢Ϝ, 2019.
[2] M. Mitsuhara, et al., “Embedding Human Knowledge
into Deep Neural Network via Attention Map”, VISAPP,
2021.