1D Self-Attention Network ʹΑΔਆ‫ܦ‬৴߸͔Βͷಈ࡞ࣝผʹؔ͢Δ‫ڀݚ‬ࢦಋ‫ڭ‬तɿ౻٢߂࿱
TR19005 ాᬒɹྏ1. ͸͡Ίʹਓ෺ͷಈ࡞ࣝผ͸ɼը૾ϕʔεͷࣝผ๏ [1] ͱ৴߸ϕʔεͷࣝผ๏ [2] ʹେผͰ͖Δɽը૾ϕʔεͷख๏͸ɼRGB Χϝϥ͔Βऔಘͨ͠ը૾Λ༻͍ΔͨΊɼ૷ண‫ܭܕ‬ଌσόΠεΛඞཁͱͤͣɼඇ઀৮Ͱಈ࡞ͷࣝผ͕ՄೳͰ͋Δɽ͔͠͠ɼ෺Λ௫ΉͳͲͷ෺ମʹରͯ͠खઌͷΈͰߦ͏ಈ࡞͸ɼΦΫϧʔδϣϯ͕ൃੜ͢Δͱࣝผ͕ࠔ೉ͱͳΔɽҰํɼ৴߸ϕʔεͷख๏͸ɼਓମʹ૷ணͨ͠σόΠε͔Β‫ిے‬৴߸Λ༻͍ΔͨΊɼΦΫϧʔδϣϯ͕ൃੜ͠ͳ͍ͱ͍͏ϝϦοτ͕͋Δɽखઌͷಈ࡞ࣝผΛߦ͏ͨΊʹ͸ɼखઌͷಈ࡞ʹਂؔ͘
ΘΔ৴߸Λ‫ܭ‬ଌ͢Δඞཁ͕͋Δɽͦ͜Ͱຊ‫Ͱڀݚ‬͸ɼਓମͷखटʹҐஔ͠ɼखͷ‫ے‬೑Λࢧ഑͢Δखࠜ؅ਆ‫͔ܦ‬Β‫ܭ‬ଌՄೳͳਆ‫ܦ‬৴߸ʹண໨͢Δɽਆ‫ܦ‬৴߸ͷ࣌‫ྻܥ‬σʔλ͔Βಈ࡞‫ݻ‬༗ͷಛ௃Λଊ͑ΔͨΊʹɼ1D-Self-Attention BlockΛఏҊ͢Δɽ⨀Pointwise conv
1×1×C’

1×1×CHSelf attention map
1×1×C’

C& % $

W
H×W×C

,"
1×1×C

1×1×C

Generated SAM
1×W×C’⨀

Pointwise conv
input

Pointwise conv

1×1×C’

1×1×C

& % $
1×W×C

Bn/Relu⨁

Bn/Relu⨁,"
1×1×C

Output

SAM
1×W×C

1×1×C

スキップ機構: 近傍時刻１(t - 1)

: 注目時刻( t ): 近傍時刻２(t + 1)

ਤ 1 : 1D Self-Attention Block ͷߏ଄

2. ैདྷ‫ڀݚ‬৴߸ϕʔεͷैདྷख๏ͱͯ͠ɼMorbidoni Β͸ද໘‫ిے‬ਤ (sEMG) Λ֊૚‫ܕ‬χϡʔϥϧωοτϫʔΫʹೖྗͯ͠ɼਓ෺ͷελϯεϑΣʔζͱεΠϯάϑΣʔζͷಈ࡞ࣝผΛߦ͏ख๏ΛఏҊ͍ͯ͠Δ [3]ɽ·ͨɼOzal Β͸ਓؒͷ৺ిਤ͔Βϑϥάϝϯτͱ‫ݺ‬͹ΕΔখྖҬΛαϯϓϦϯά͠ɼ
CNN ΛదԠ͢Δ͜ͱͰ 17 Ϋϥεͷ৺ෆશΛ‫ݕ‬ग़͢Δख๏ΛఏҊ͍ͯ͠Δ [2]ɽ͜ΕΒͷैདྷख๏͸ɼΫϥεࣝผͷͨΊͷಛ௃நग़ʹଟ૚ύʔηϓτϩϯ΍৞ΈࠐΈχϡʔϥϧωοτϫʔΫΛ༻͍͍ͯΔɽ͔͠͠ɼ৞ΈࠐΈ૚͸ɼۙ๣ͷ࣌ࠁʹண໨͢ΔͨΊɼ཭Εͨ࣌ࠁͷॏཁͳมԽΛଊ͑Δ͜ͱʹద͍ͯ͠ͳ͍ɽ

3. ఏҊख๏

3.3. ωοτϫʔΫͷߏ଄ຊ‫Ͱڀݚ‬͸ɼ̏૚ͷ 1D-SAB ͔ΒͳΔ 1D-SAN Λ༻͍ΔɽҰ૚໨ͷ 1D-SAB ʹ͓͚Δۙ๣࣌ࠁͷִؒΛ 3 ࣌ࠁɼ
2ɼ3 ૚໨ͷ 1D-SAB ʹ͓͚Δۙ๣࣌ࠁͷִؒΛ 5 ࣌ࠁͱ͢Δɽ͜ΕʹΑΓɼ཭Εͨ࣌ࠁͷ৴߸ͱͷมԽΛߟྀͰ͖Δɽ
1D-SAN ͷߏ଄ͱ֤ 1D-SAB ͷۙ๣ઃఆΛਤ 2 ʹࣔ͢ɽt-3

inputt+3

3.1. खࠜ؅ਆ‫ܦ‬ຊ‫Ͱڀݚ‬͸ɼਓ෺ͷखઌͷಈ࡞Λର৅ͱ͢ΔͨΊɼਓମͷखटʹҐஔ͠ɼखͷ‫ے‬೑Λࢧ഑͢Δखࠜ؅ਆ‫͔ܦ‬Β‫ܭ‬ଌՄೳͳਆ‫ܦ‬৴߸Λ༻͍Δɽखࠜ؅ਆ‫ʹܦ‬͸ɼᒷࠎਆ‫ܦ‬(Radial narve)ɼਖ਼தਆ‫( ܦ‬Median narve)ɼईࠎਆ‫( ܦ‬Ulnar narve) ͕͋Δɽ͜ΕΒͷਆ‫ܦ‬৴߸Λ Mudra wearabledevice ʹΑΓ‫ܭ‬ଌ͢ΔɽαϯϓϦϯάִؒ͸ 1ms ͱ͢Δɽ

3.2.1D Self-Attention Networks(1D-SAN)ຊ‫Ͱڀݚ‬͸ɼSelf-Attention Networks(SAN)[4] Ͱ༻͍ΒΕΔ Self-Attention Block(SAB) Λ 1 ࣍‫ݩ‬σʔλʹରԠͤͨ͞ɼ1D-SAB ΛఏҊ͠ɼ1D-SAN Λߏ੒͢ΔɽఏҊ
͢Δ 1D-SAB ͷߏ଄Λਤ 1 ʹࣔ͢ɽ1D-SAB ʹ͸ɼखࠜ؅ਆ‫͔ܦ‬Βऔಘͨ࣌͠‫ྻܥ‬৴߸Λೖྗ͢Δɽೖྗͨ͠৴߸
͸ɼ̍࣌ࠁ͝ͱʹॲཧ͠ɼରԠ͢Δ࣌ࠁͷ Self-AttentionΛࢉग़͢Δɽਤ 1 ʹ͓͚Δ੨৭ͷ஋Λॲཧͷ஫໨࣌ࠁͱ͠
ͨ࣌ɼ੺৭Λۙ๣࣌ࠁ̍, ྘৭Λۙ๣࣌ࠁ̎ͱ͢Δɼ֤ۙ๣࣌ࠁʹରͯ͠ɼ஫໨࣌ࠁ (੨) ͱͷ Pointwise conv ॲཧΛߦ͏ɽ·ͨɼ஫໨࣌ࠁͱۙ๣࣌ࠁΛɼП͓ΑͼСͷֶशՄೳͳؔ਺ (1D-CNN) ΁ೖྗ͠ɼؔ܎ؔ਺ЎΛ‫ٻ‬ΊΔɽؔ܎ؔ਺Ўͷग़ྗ͸ɼؔ਺Пͷग़ྗ͔Βؔ਺Сͷग़ྗΛ‫ࢉݮ‬ͨ͠΋ͷͰ͋ΓɼϚοϐϯάؔ਺ЍʹΑͬͯ̍ͭ໨ͷॲཧͷग़ྗͱνϟϯωϧ਺Λ߹ΘͤΔɽͦͷ‫ޙ‬ɼ2 ͭͷग़ྗͷཁૉੵΛࢉग़͢Δɽ͜ͷॲཧΛۙ๣࣌ࠁ෼ߦ͍૯࿨͢Δɽ
ੜ੒ͨ͠ Self-Attention Map(SAM) ͸ Pointwise conv ॲཧʹΑΓೖྗνϟωϧ਺ͱಉ͡νϟωϧʹ͢Δɽ͜ͷग़ྗʹɼεςοϓ‫ͯ͠ͱߏػ‬ೖྗ৴߸ΛՃࢉ͠ɼ࠷ऴग़ྗͱ͢
Δɽ1D-SAB Λ༻͍Δ͜ͱͰɼਆ‫ܦ‬৴߸಺ͷॏཁͳҐஔʹେ͖ͳॏΈΛ༩͑Δ͜ͱ͕Ͱ͖ɼ཭Εͨ৴߸஋ͱͷؔ܎ੑΛߟྀͰ͖Δɽ

t-5

1D-SAB1ਓ෺ͷखઌͷಈ࡞͸ɼͦͷछྨʹΑΓಈ࡞͕࣌ؒҟͳΔɽͦͷͨΊɼಈ࡞ͷ୹‫ظ‬త͓Αͼ௕‫ظ‬తͳมԽΛଊ͑Δࣄ͕ॏ
ཁͱͳΔɽຊ‫Ͱڀݚ‬͸ɼ࣌‫ྻܥ‬σʔλʹ͓͍ͯɼपล࣌ࠁΛߟྀͨ͠ಛ௃நग़͕Մೳͳ 1D-Self-Attention Block(1DSAB) ʹΑΓߏ੒͞ΕΔ 1D-Self-Attention Networks(1DSAN) Λ༻͍ͨಈ࡞ࣝผ๏ΛఏҊ͢Δɽ

FCtt

1D-SAB2

1D-SAB3・・・・・tt-5t+5・・・・・t+5・・・6 classਤ 2 : ωοτϫʔΫߏ଄ͱۙ๣

3.4. ಈ࡞ࣝผͷྲྀΕຊ‫Ͱڀݚ‬ͷಈ࡞ࣝผ͸ɼ3 ͭͷ Step ͱͳΔɽStep 1 ͸ɼ
Mudra wearable device Λ༻͍ͯऔಘͨ͠ਆ‫ܦ‬৴߸ΛҰఆ۠ؒ͝ͱʹ‫ݻ‬ఆ௕ͰαϯϓϦϯά͠ɼ1D-SAN Ͱਪ࿦͕Ͱ
͖Δσʔλ΁੔‫͢ܗ‬ΔɽStep 2 ͸ɼֶशࡁΈ 1D-SAN ʹɼࣝผର৅ͷσʔλͷ 0 ࣌ࠁΛ‫ͯ͠ͱ఺ى‬ɼೖྗ͢ΔσʔλͷྖҬΛ 10 ࣌ࠁͣͭ‫܁‬Γฦ͠ೖྗ͠ɼಈ࡞ࣝผ͢ΔɽStep3 ͸ɼ֤۠ؒͷಈ࡞ࣝผ݁ՌΛ౷߹ͯ͠࠷ऴతͳಈ࡞ࣝผ݁ՌΛग़ྗ͢Δɽ

Step 1ɿલॲཧຊ‫Ͱڀݚ‬ѻ͏ਆ‫ܦ‬৴߸೾‫ܗ‬͸ɼՄม௕ͷσʔλͰ͋Δɽ
1D-SAN Ͱਪ࿦Λߦ͏ͨΊʹɼ64 ࣌ࠁ෼ͷ 1 ࣍‫ݩ‬೾‫ܗ‬ΛαϯϓϦϯά͢Δɽ

Step 2ɿαϯϓϦϯά࣌ࠁͰͷಈ࡞ࣝผಈ࡞ࣝผ͸ɼ10 ࣌ࠁͣͭαϯϓϦϯά۠ؒΛͣΒ͠ͳ͕Β 1D-SAN ʹೖྗ͢Δɽ

Step 3ɿಈ࡞ࣝผ݁Ռͷ౷߹֤࣌ࠁʹ͓͚ΔωοτϫʔΫͷग़ྗΛΫϥεຖʹྦྷੵ͢Δɽྦྷੵ஋͕࠷େͷΫϥεΛಈ࡞ࣝผ݁Ռͱ͢Δɽࣝผ݁ՌྦྷੵॲཧΛਤ 3 ʹࣔ͢ɽ

4. ධՁ࣮‫ݧ‬ఏҊख๏ͷ༗ޮੑΛࣔͨ͢Ίʹɼ1 ࣍‫ݩ‬೾‫ʹܗ‬ରԠͨ͠৞ΈࠐΈχϡʔϥϧωοτϫʔΫ (1D-CNN) ͱೝࣝਫ਼౓ͷൺֱΛߦ͏ɽ·ͨɼϑʔϦΤม‫͠׵‬ɼप೾਺ۭؒΛೖྗͱͨ͠ࡍͷਫ਼౓ͱ΋ൺֱ͢Δɽ

4.1. σʔληοτຊ‫Ͱڀݚ‬͸ɼ6 Ϋϥεͷಈ࡞ʹର͢Δਆ‫ܦ‬৴߸Λ 100 ճ‫ܭ‬ଌͨ͠ 600 αϯϓϧΛ༻͍Δɼ͜ΕΒΛ 6:4 Ͱֶश͓ΑͼධՁσʔλʹ෼ׂͯ͠༻͍ΔɽΫϥεͷσʔλऔಘ࣌ͷ

・・・

t=1

t=2

input

1D-SAN

t=n

1D-SAN

output・・・

1D-SAN

6クラス分の識別結果[!!!, !$!, !%!, !&!, !'!, !(!],[!!$, !$$, !%$, !&$, !'$, !($],要素和・・・ [!!#, !$#, !%#, !&#, !'#, !(#]

͸‫ݮ‬গͨ͠ɽҰํͰఏҊख๏͸ɼPush Λ Twist ͱ‫ࣝޡ‬ผ͢Δ͜ͱ͕ଟ͍ɽPush ʹରͯ͠ఏҊख๏ͷਫ਼౓͕௿Լͨ͠ཁҼ͸ɼ෺Λԡ͠ࠐΉಈ࡞Λ͢ΔࡍʹɼྗΛࠐΊͯ‫͘ڧ‬ԡ͠ࠐΉέʔε͕͋ΓɼPush Ͱ͋Γͳ͕Β Twist ͷΑ͏ʹৼಈ͢Δ৴߸͕‫ܭ‬ଌ͞Εͨ͜ͱ͕‫ݪ‬Ҽͱߟ͑ΒΕΔɽ͜ͷΑ͏ͳέʔεΛֶशσʔλʹ΋௥Ճ͢Δඞཁ͕͋Δɽ·ͨɼTurn ΍ Twist ͳͲɼ࣌‫ྻܥ‬తͳมԽ͕େ͖ͳΫϥεʹର͢Δਫ਼౓͕޲্͍ͯ͠Δ͜ͱ͔ΒɼఏҊख๏͸৴߸஋͕࣌‫ྻܥ‬తʹେ͖͘มԽ͢ΔΫϥεʹ༗ޮͰ͋Δɽ

最終識別結果argmax( ∑ #" "!", ∑ #" "$", ∑ #" "%", ∑ #" "&" , ∑ #" "'" , ∑ #" "(" )ද 1 : Confusion Matrix ʹΑΔࣝผ݁Ռͷൺֱ

ਤ 3 : ࠷ऴॲཧ൑ఆը૾Λਤ 4 ʹࣔ͢ɽσʔλ͸ɼMudra wearable device Λඃ‫ऀݧ‬ͷӈखटʹ૷ண͠ɼಈ࡞தͷ 1 ࣍‫ݩ‬೾‫ܗ‬Λऩूͨ͠ɽ
ਤ 5 ʹಈ࡞ Push ʹର͢Δਆ‫ܦ‬৴߸Λࣔ͢ɽ

PushTwistTakeTypeද 2 ʹɼप೾਺ྖҬʹ͓͚Δ֤ख๏ͷਫ਼౓Λࣔ͢ɽఏҊख๏ͱ 1D-CNN ͰൺֱΛߦͬͨ݁Ռɼप೾਺ྖҬͰ͸ɼ
1D-SAN ͷಈ࡞ࣝผਫ਼౓͕ 80.0 ˋɼ1D-CNN ͷಈ࡞ࣝผਫ਼౓͕ 90.4 ˋͱͳΓɼ1D-SAN ͷਫ਼౓͕ԼճͬͨɽҰํͰϑʔϦΤม‫Ͱ͠ͳ׵‬͸ɼ1D-SAN ͷࣝผਫ਼౓͸ 91.2 ˋͱ޲্ͨ͠ɽ1D-SAN ͸ɼ૬ରతͳҐஔؔ܎ͱͦͷॏཁੑ
Λ Self-Attention ʹΑΓද‫͖Ͱݱ‬ΔͨΊϑʔϦΤม‫͠ͳ׵‬Ͱಈ࡞ࣝผ͕ՄೳͰ͋Δɽ
ද 2 : प೾਺ྖҬʹ͓͚Δਫ਼౓ൺֱ

TurnPut

ਤ 4 : ର৅ಈ࡞ΫϥεωοτϫʔΫ

ϑʔϦΤม‫͋׵‬Γ (%)ϑʔϦΤม‫( ͠ͳ׵‬%)1D-SAN

80.0

91.2

1D-CNN

90.4

90.4

5. ͓ΘΓʹ
ຊ‫Ͱڀݚ‬͸ɼखઌͷ‫ے‬೑Λࢧ഑͢Δखࠜ؅ਆ‫͔ܦ‬Βऔಘͨ͠ਆ‫ܦ‬৴߸͔Βͷಈ࡞ࣝผΛߦ͏ͨΊʹɼ1 ࣍‫ݩ‬೾‫ʹܗ‬ରԠͨ͠ 1D Self-Attention Network ΛఏҊͨ͠ɽਫ਼౓ൺֱ࣮‫Ͱݧ‬͸ɼैདྷख๏ͷ 1D-CNN ʹରͯ͠ɼఏҊख๏͕
໿ 0.8 ϙΠϯτ্ճΓɼಈ࡞ࣝผʹ͓͍ͯɼఏҊख๏͕༗
ޮͰ͋Δ͜ͱΛࣔͨ͠ɽ·ͨɼPush ͷΑ͏ͳਆ‫ܦ‬৴߸஋͕࿈ଓతʹେ͖͘ͳΔΫϥεͰ͸ɼਫ਼౓͕௿Լ͕ͨ͠ɼTurn΍ Twist ͳͲɼ࣌‫ྻܥ‬తͳมԽ͕ଊ͑΍͍͢Ϋϥεʹؔͯ͠ਫ਼౓͕޲্ͨ͠ɽࢀߟจ‫ݙ‬

ਤ 5 : ৴߸σʔλαϯϓϧ (Push)

4.2. ࣮‫֓ݧ‬ཁ
1D-CNN Λ༻͍ͨϞσϧͱ 1D-SAB Λ༻͍ͨϞσϧͷਫ਼౓ൺֱΛߦ͏ɽ1D-CNN ͸ɼ2 ૚ͷ৞ΈࠐΈ૚ͱग़ྗ૚ͷ‫ ܭ‬3 ૚ͷߏ଄Ͱ͋Δɽֶश͸ɼEpoch ਺Λ 20000ɼ
BatchSize Λ 128ɼαϯϓϦϯά෯Λ 64 ͱ͢ΔɽOptimizer͸ Momentum SGD ͱ͠ɼֶश཰Λ 0.001 ɼLoss ؔ਺ʹ
CrossEntropy Loss Λ༻͍Δɽ

4.3. ࣮‫݁ݧ‬Ռ

ද 1 ʹ֤ख๏ͷ Confusion Matrix Λࣔ͢ɽൺֱ࣮‫ݧ‬ͷ݁Ռɼ1D-CNN ͷಈ࡞ࣝผਫ਼౓͸ 90.4 ˋɼఏҊख๏͸
91.2 ˋͱͳΓɼఏҊख๏͕ಈ࡞ࣝผͰ༗ޮͰ͋Δ͜ͱ͕Θ͔ͬͨɽఏҊख๏͸ɼTurnɼTwistɼType ͷਫ਼౓͕޲্
͠ɼTakeɼPush ͷਫ਼౓͕௿Լͨ͠ɽ1D-CNN Ͱ͸ɼTwistΛ Turn ͱ‫ࣝޡ‬ผ͕ͨ͠ɼఏҊख๏Ͱ͸ɼTurn ΁ͷ‫ࣝޡ‬ผ

[1] T. Xiao,et al., “Reasoning About Human-Object
Interactions Through Dual Attention Networks”,ICCV, 2019.
[2] O. Yildirim, et. al. , “Arrhythmia detection using
deep convolutional neural network with long duration ECG signals”, Computers in Biology andMedicine, vol102, 2018, p411 420.
[3] C.Moribidoni,et al., “A Deep Learning Approachto EMG-Based Classification of Gait Phases during
Level Ground Walking” ,MDPI, 2019.
[4] Hengshuang.Zhao,et al., “Exploring Self attention
for Image Recognition” ,CVPR, 2020.

‫੷ۀڀݚ‬[1] ాᬒྏ ౳, “Self-Attention Networks ʹΑΔਆ‫ܦ‬৴߸͔Βͷಈ࡞ࣝผ”, Ϗδϣϯٕज़ͷ࣮ར༻ϫʔΫγϣοϓ, 2020.