ۭ࣌ؒͷΞςϯγϣϯ৘ใΛߟྀͨ͠ Graph Convolutional Networks ʹΑΔಈ࡞ೝࣝʹؔ͢Δ‫ڀݚ‬ࢦಋ‫ڭ‬तɿࢁԼོٛ

TP19008 ന໦ࠀढ़2. ࠎ֨σʔλ͔Βͷ GCN Λ༻͍ͨಈ࡞ೝࣝࠎ֨σʔλ͔Βͷಈ࡞ೝࣝʹ, GCN Λ༻͍ͨख๏Ͱ͋ΔSpatial Temporal GCN (ST-GCN) [1] ͕͋Δ. ST-GCN
͸, ಉҰϑϨʔϜ಺ͷؔઅΛ݁Ϳۭؒάϥϑͱ, ྡ઀ϑϨʔϜͷಉҰؔઅΛ݁Ϳ࣌ؒάϥϑͷ 2 ͭͷάϥϑߏ଄ (ਤ 1)Ͱࠎ֨σʔλΛද‫͢ݱ‬Δ͜ͱʹΑΓߴ͍ೝࣝਫ਼౓Λୡ੒͠
ͨ. ͔͠͠ͳ͕Β, ST-GCN ͸, ۭؒάϥϑͷ઀ଓؔ܎ ESΛࠎ֨ύλʔϯ H Ͱఆ͍ٛͯ͠ΔͨΊ, ಈ࡞ಛ༗ͷؔઅؒͷؔ܎ੑΛߟྀͰ͖ͳ͍. ·ͨ, ಈ࡞΍࣌ؒ͝ͱʹมԽ͢Δؔઅͷॏཁ౓΋ද‫͏͍ͱ͍ͳ͖Ͱݱ‬໰୊͕͋Δ.

時間グラフ： EF = {vti , v(t 1)i }<latexit sha1_base64="WO6MoIs0XDy6Os0H0DA6rik5Ljk=">AAACgXichVFNS+RAEH1G1x2jq6NeBD3IDoqCO1REUFwEURSPfo0KjoQk27qNmSQkPQOzYS573D/gwZOCqHhy/8Je/AMe/AmLRwUvHqzJBBYV1wrpev26XvXrbjtwZaSIbpq05pYPrR8zbXp7x6fOrmx3z0bkl0NHFBzf9cMt24qEKz1RUFK5YisIhVWyXbFp78/X1zcrIoyk762raiB2StaeJ3elYymmzOzAgrk4oxdjvWLGStbGOI2oL8aorOnFmpnNUZ6SGHwNjBTkkMaynz1FEd/gw0EZJQh4UIxdWIj424YBQsDcDmLmQkYyWReoQWdtmasEV1jM7vO4x7PtlPV4Xu8ZJWqHd3H5D1k5iCG6pnO6oyu6oL/0+GavOOlR91LlbDe0IjC7fvWtPbyrKnFW+P5P9V/PCruYSrxK9h4kTP0UTkNf+XFwtza9OhQP0zHdsv8juqE/fAKvcu+crIjVQ+j8AMbL634NNsbzBuWNlYnc7Fz6FBn04zNG+L4nMYslLKPA+/7EGS7xW2vWRjXSxhulWlOq6cWz0L4+AR5fkuQ=</latexit>#：骨格パターン!：フレーム"：ノード数

ਤ 1: ST-GCN ʹ͓͚Δࠎ֨σʔλͷάϥϑද‫ݱ‬

3. ఏҊख๏
ؔઅͷॏཁ౓ͱؔ܎ੑΛߟྀͯ͠ಈ࡞ೝࣝΛߦ͏ Spatial Temporal Attention GCN (STA-GCN) ΛఏҊ͢Δ.
STA-GCN ͸ؔઅͷॏཁ౓Ͱ͋Δ Attention node ͱ, ؔઅؒͷؔ܎ੑͰ͋Δ Attention edge Λ֫ಘ͢Δ. ֫ಘͨ͠ Attention Λߟྀͭͭ͠ೝࣝΛߦ͏. ·ͨ, ϚϧνϞʔμϧֶशʹ͓͍ͯ Mechanics-stream ߏ଄ΛఏҊ͢Δ. Mechanicsstream ߏ଄͸, ֤Ϟʔμϧ͕࣋ͭྗֶతಛੑ΍஋ͷεέʔϧͷҧ͍ʹ΋ͱ͍ͮͯωοτϫʔΫΛߏங͢Δ.

3.1 STA-GCN ͷωοτϫʔΫߏ଄
ਤ 2 ʹ STA-GCN ͷωοτϫʔΫߏ଄Λࣔ͢. STAGCN ͸, Feature extractor, Attention branch, Perception branch ͷ 3 ͭͷϞδϡʔϧͰߏ੒͢Δ. ֤Ϟδϡʔϧʹ‫·ؚ‬ΕΔ STGC-block Ͱ͸, ۭؒάϥϑͱ࣌ؒάϥϑͷάϥϑ৞ΈࠐΈॲཧΛߦ͏. Feature extractor ʹ͸ 2 ͭͷϞʔμϧΛೖྗ͠, ਓؒͷࠎ֨ύλʔϯ Askeleton Λۭؒάϥϑͱ͢Δෳ਺ͷ STGC-block ʹΑΓ֤Ϟʔμϧͷಛ௃ϚοϓΛ֫ಘ͠, ݁߹͢Δ. Attention branch ͸, ؔઅͷॏཁ౓Λද͢ Attention node ͱ, ؔઅؒͷॏཁͳؔ܎ੑΛද͢ Attention edge Λಛ௃Ϛοϓ͔Β֫ಘ͢Δ. Perceptionbranch ͸, Attention node ͱ Attention edge Λߟྀͭͭ͠, ࠷ऴతͳΫϥε֬཰Λग़ྗ͢Δ. ֶश͸, 2 ͭͷϒϥϯν͔Βͷग़ྗʹରͯ͠ΫϩεΤϯτϩϐʔ‫Ͱࠩޡ‬ଛࣦΛ‫ٻ‬Ί, ଛࣦͷ࿨ΛωοτϫʔΫͷֶश‫͍༻ͯ͠ͱࠩޡ‬Δ.

Batch
Normalization

FCSoftmaxFC

Softmax

1 1 conv.
1 1 conv.

GAP

TanhBatchNormalization

ReLU

1 1 conv.

GAP

Latt

Lper

<latexit sha1_base64="rzLIR/W4Zap6hMNhKgk0nYv5i30=">AAACanichVG7SgNBFD1ZXzE+EmOjpBGjYhXuiqBYCTYWFiYxRlAJu+uoSza7y+4kEEN+wM5K0EpBRPwMG3/AIp8g2inYWHizWRAV9Q4zc+bMPXfOzOiuZfqSqBVRurp7evui/bGBwaHheGIkuek7Vc8QBcOxHG9L13xhmbYoSFNaYsv1hFbRLVHUyyvt/WJNeL7p2Buy7ordinZgm/umoUmmimulhiZls5RIU4aCmPgJ1BCkEca6k7jGDvbgwEAVFQjYkIwtaPC5bUMFwWVuFw3mPEZmsC/QRIy1Vc4SnKExW+bxgFfbIWvzul3TD9QGn2Jx91g5gWl6oBt6oXu6pUd6/7VWI6jR9lLnWe9ohVuKH4/l3/5VVXiWOPxU/elZYh+LgVeTvbsB076F0dHXjk5f8ku56cYMXdIT+7+gFt3xDezaq3GVFblzxPgD1O/P/RNszmVUyqjZ+fRyLvyKKFKYxCy/9wKWsYp1FAJ3JzjDeeRZSSrjSqqTqkRCzSi+hDL1AcvijLg=</latexit>

Askeleton

STGC-Block 5<latexit sha1_base64="aYu0bwSDva65Ct14jn/gNrFJXWM=">AAACd3ichVHLSsNAFD2N7/qquhFcWCyKq3IjguJKcePStlYFHyWJUw1Nk5BMixr6A/6AC0FQkCp+hht/wEU/QVwqiODC2zQgKuodZubMmXvunJnRXcv0JVEjprS1d3R2dffEe/v6BwYTQ8PrvlPxDJE3HMvxNnXNF5Zpi7w0pSU2XU9oZd0SG3ppubm/URWebzr2mjxyxU5Z27fNomlokqlCYjjY1ovJpdpu4JeEJaRj1wqJFKUpjORPoEYghShWnUQd29iDAwMVlCFgQzK2oMHntgUVBJe5HQTMeYzMcF+ghjhrK5wlOENjtsTjPq+2ItbmdbOmH6oNPsXi7rEyiUl6oBt6pnu6pUd6/7VWENZoejniWW9phVsYPBnNvf6rKvMscfCp+tOzRBHzoVeTvbsh07yF0dJXj0+fcwvZyWCKLumJ/V9Qg+74Bnb1xbjKiOwZ4vwB6vfn/gnWZ9IqpdXMbGoxG31FN8YwgWl+7zksYgWryPO5hzhHHdexN2VcmVKmW6lKLNKM4Eso6ge6I5Gj</latexit>

Perception branch

X%&"#$

STGC-Block 5X/"#$

AskeletonSTGC-Block 5<latexit sha1_base64="aYu0bwSDva65Ct14jn/gNrFJXWM=">AAACd3ichVHLSsNAFD2N7/qquhFcWCyKq3IjguJKcePStlYFHyWJUw1Nk5BMixr6A/6AC0FQkCp+hht/wEU/QVwqiODC2zQgKuodZubMmXvunJnRXcv0JVEjprS1d3R2dffEe/v6BwYTQ8PrvlPxDJE3HMvxNnXNF5Zpi7w0pSU2XU9oZd0SG3ppubm/URWebzr2mjxyxU5Z27fNomlokqlCYjjY1ovJpdpu4JeEJaRj1wqJFKUpjORPoEYghShWnUQd29iDAwMVlCFgQzK2oMHntgUVBJe5HQTMeYzMcF+ghjhrK5wlOENjtsTjPq+2ItbmdbOmH6oNPsXi7rEyiUl6oBt6pnu6pUd6/7VWENZoejniWW9phVsYPBnNvf6rKvMscfCp+tOzRBHzoVeTvbsh07yF0dJXj0+fcwvZyWCKLumJ/V9Qg+74Bnb1xbjKiOwZ4vwB6vfn/gnWZ9IqpdXMbGoxG31FN8YwgWl+7zksYgWryPO5hzhHHdexN2VcmVKmW6lKLNKM4Eso6ge6I5Gj</latexit><latexit sha1_base64="z3Ou31v8Q/t/Y9rkeF3QiqHVQJk=">AAACanichVG7SgNBFD1ZXzE+EmOjpBGjYhXuiqBYBWwsLPIwRlAJu+uoSza7y+4mEEN+wM5KMJWCiPgZNv6ART5BtItgY+HNZkFU1DvMzJkz99w5M6Pahu56RO2Q1Nc/MDgUHo6MjI6NR2MT8W3XqjqaKGiWYTk7quIKQzdFwdM9Q+zYjlAqqiGKanm9u1+sCcfVLXPLq9tiv6IcmfqhrikeU8XNUsMWTrMUS1KK/Jj5CeQAJBFExordYA8HsKChigoETHiMDShwue1CBsFmbh8N5hxGur8v0ESEtVXOEpyhMFvm8YhXuwFr8rpb0/XVGp9icHdYOYN5eqRb6tAD3dETvf9aq+HX6Hqp86z2tMIuRU+n8m//qio8ezj+VP3p2cMhVn2vOnu3faZ7C62nr52cd/JrufnGAl3RM/u/pDbd8w3M2qt2nRW5FiL8AfL35/4JtpdSMqXk7HIynQu+IowEZrHI772CNDaQQcF3d4YLtEIvUlyalhK9VCkUaCbxJaS5D8fvjLY=</latexit>

Aattention , Askeleton<latexit sha1_base64="cBh/6QephkG1NQXqYsZzQlLsuMQ=">AAACjHicSyrIySwuMTC4ycjEzMLKxs7BycXNw8vHLyAoFFacX1qUnBqanJ+TXxSRlFicmpOZlxpaklmSkxpRUJSamJuUkxqelO0Mkg8vSy0qzszPCympLEiNzU1Mz8tMy0xOLAEKxQuoV8ckpSk41sZVJ5aUpOaBBGt1FOCCxdmpOaklQLF4AWUDPQMwUMBkGEIZygxQEJAvsJwhhiGFIZ8hmaGUIZchlSGPoQTIzmFIZCgGwmgGQwYDhgKgWCxDNVCsCMjKBMunMtQycAH1lgJVpQJVJAJFs4FkOpAXDRXNA/JBZhaDdScDbckB4iKgTgUGVYOrBisNPhucMFht8NLgD06zqsFmgNxSCaSTIHpTC+L5uySCvxPUlQukSxgyELrwurmEIY3BAuzWTKDbC8AiIF8kQ/SXVU3/HGwVpFqtZrDI4DXQ/QsNbhocBvogr+xL8tLA1KDZDFzACDBED25MRpiRnqGBnmGgibJDEDQqOBikGZQYNIDhbc7gwODBEMAQCrS3l2Ezwx6GvUx8TCZM1ky2EKVMjFA9wgwogMkNAHa1mWs=</latexit>

ਤ 2: Spatial Temporal Attention GCNAttention edge ͸, ಈ࡞͝ͱͷؔઅؒͷॏཁͳؔ܎ੑΛද͢ྡ઀ߦྻͰ͋Δ. ֫ಘͨ͠ Attention edge ͸ Perception
branch ʹ༩͑Δ. Perception branch Ͱ͸, Attention edge
Aattention ͱਓؒͷࠎ֨ύλʔϯ Askeleton ʹΑΔάϥϑ
৞ΈࠐΈॲཧΛߦ͏. Perception branch ʹ͓͚Δೖྗಛ௃ Xin ʹର͢Δۭؒάϥϑ৞ΈࠐΈॲཧΛࣜ (2) ʹࣔ͢.
Mskel , Wskel , Watt ͸ॏΈߦྻͰ͋Δ. Q ͸ hop ਺Ͱ͋Γ Q ‫཭ݸ‬Εͨؔઅ·ͰΛ݁Ϳ. K ͸ಈ࡞͝ͱʹੜ੒͢ΔAttention edge ͷ਺Ͱ͋Δ. ຊ‫Ͱڀݚ‬͸ Q = 3, K = 4 ͱ͢Δ.
Q!
Xout =Mskel◦ Âskel
Xin Wqskelqqq+<latexit sha1_base64="045F/2C3Ksius+g4s7VfI3ujK8o=">AAACjHicSyrIySwuMTC4ycjEzMLKxs7BycXNw8vHLyAoFFacX1qUnBqanJ+TXxSRlFicmpOZlxpaklmSkxpRUJSamJuUkxqelO0Mkg8vSy0qzszPCympLEiNzU1Mz8tMy0xOLAEKxQuoh9nGVHOVxVeXZNbWlNga6sSk5JcU64ToKGTCOX5cMbXxAsoGegZgoIDJMIQylBmgICBfYDlDDEMKQz5DMkMpQy5DKkMeQwmQncOQyFAMhNEMhgwGDAVAsViGaqBYEZCVCZZPZahl4ALqLQWqSgWqSASKZgPJdCAvGiqaB+SDzCwG604G2pIDxEVAnQoMqgZXDVYafDY4YbDa4KXBH5xmVYPNALmlEkgnQfSmFsTzd0kEfyeoKxdIlzBkIHThdXMJQxqDBditmUC3F4BFQL5Ihugvq5r+OdgqSLVazWCRwWug+xca3DQ4DPRBXtmX5KWBqUGzGbiAEWCIHtyYjDAjPUMDPcNAE2UHJ2hUcDBIMygxaADD25zBgcGDIYAhFGhvL8Nmhj0Me5n4mEyYrJlsIUqZGKF6hBlQAJMbAPxCly4=</latexit><latexit sha1_base64="Agw0HGDAh3iEqbKTurGdvpX284A=">AAACiHicSyrIySwuMTC4ycjEzMLKxs7BycXNw8vHLyAoFFacX1qUnBqanJ+TXxSRlFicmpOZlxpaklmSkxpRUJSamJuUkxqelO0Mkg8vSy0qzszPCympLEiNzU1Mz8tMy0xOLAEKxQsou8YH23LFVHOVxVeXZNbqgKis2hqNTJ0szZjMPAUPrphaoCoDPQMwUMBkGEIZygxQEJAvsJwhhiGFIZ8hmaGUIZchlSGPoQTIzmFIZCgGwmgGQwYDhgKgWCxDNVCsCMjKBMunMtQycAH1lgJVpQJVJAJFs4FkOpAXDRXNA/JBZhaDdScDbckB4iKgTgUGVYOrBisNPhucMFht8NLgD06zqsFmgNxSCaSTIHpTC+L5uySCvxPUlQukSxgyELrwurmEIY3BAuzWTKDbC8AiIF8kQ/SXVU3/HGwVpFqtZrDI4DXQ/QsNbhocBvogr+xL8tLA1KDZDFzACDBED25MRpiRnqGBnmGgibKDEzQqOBikGZQYNIDhbc7gwODBEMAQCrS3k2E9ww6GnUxcTAZM5kyWEKVMjFA9wgwogMkJAATplfA=</latexit>

STGC-Block 5Feature extractor

ノード：V = {vti |t = 1, . . . , T, i = 1, . . . , N }空間グラフ： ES = {vti , vtj |(i, j) 2 H}

Batch
Normalization

Attention node '(X %&"#$ )Sigmoidࠎ֨σʔλ͸ਓؒͷಈ͖Λ௚઀ଊ͑Δ͜ͱ͕Ͱ͖Δ͜ͱ΍, ೝࣝ࣌ʹ͓͚Δ‫ڥ؀‬΍ࢹ఺ͷมԽʹରͯ͠‫ͳ݈ؤ‬ར఺͕͋Δ. ࠎ֨σʔλΛ༻͍ͨ Graph Convolutional Networks(GCN) ʹΑΔैདྷͷಈ࡞ೝࣝͰ͸, ਓؒͷࠎ֨ύλʔϯͰάϥϑߏ଄Λࣄલʹఆٛ͢ΔͨΊ, ಈ࡞ಛ༗ͷؔઅؒͷؔ܎ੑΛߟྀͰ͖ͳ͍. ·ͨ, ೝࣝʹ͓͚Δؔઅͷॏཁ౓΋ಈ࡞͝ͱʹҟͳΔ͜ͱ͕༧૝͞ΕΔ. ຊ‫Ͱڀݚ‬͸, ؔઅͷॏཁ౓ͱؔ܎ੑΛߟྀͯ͠ಈ࡞ೝࣝΛߦ͏ Spatial Temporal
Attention Graph Convolutional Networks (STA-GCN)
ΛఏҊ͢Δ. STA-GCN ͸, ϑϨʔϜ͝ͱͷؔઅͷॏཁ౓ͱ, ಈ࡞͝ͱͷؔઅؒͷ઀ଓؔ܎Λ֫ಘ͢Δ. ॏཁ౓Λಛ௃ϚοϓʹॏΈ෇͚͠, ઀ଓؔ܎Λ༻͍ͯάϥϑ৞ΈࠐΈॲཧΛߦ͏͜ͱͰ, ॏཁ౓ͱؔ܎ੑΛߟྀͭͭ͠ೝࣝΛߦ
͏. ·ͨ, ϚϧνϞʔμϧֶशʹ͓͍ͯ Mechanics-streamߏ଄Λಋೖ͠, ߴਫ਼౓ԽΛ࣮‫͢ݱ‬Δ.

Attention branch

Upsampling

Attention edge A+$$,-$."-GAP

1. ͸͡ΊʹK!

(2)

attÂattk Xin WkkAttention node ͱ Attention edge ʹΑͬͯ࣌ؒత, ۭؒతͳಛ௃Λ‫ڧ‬ௐͰ͖, 2 ͭͷ Attention Λ૊Έ߹Θͤͨάϥϑද‫ݱ‬Λ Spatial Temporal Attention Graph (Attentiongraph) ͱ‫Ϳݺ‬.

3.3 Mechanics-stream ߏ଄
Mechanics-stream ߏ଄ (ਤ 3) ͸, ϚϧνϞʔμϧֶशʹ͓͚ΔωοτϫʔΫߏ଄Ͱ͋Δ. ຊ‫Ͱڀݚ‬͸, ؔઅ࠲ඪ͔Βؔઅ଎౓, ؔઅՃ଎౓, ࠎ৘ใ, ࠎ଎౓, ࠎՃ଎౓Λࢉग़͠ωοτϫʔΫͷೖྗͱ͢Δ. ࠎ৘ใ͸, ؔઅؒͷ‫཭ڑ‬Ͱ͋Γؔઅؒͷํ޲΋ද‫͢ݱ‬Δ. ࠲ඪ΍ࠎ৘ใ͸ۭؒతͳҐஔͷ৘ใ, ଎౓΍Ճ଎౓͸࣌ؒతͳҠಈྔͷ৘ใͰ͋Γ,֤Ϟʔμϧ͕࣋ͭಛੑʹ͸ҧ͍͕͋Δ. ·ͨ, ࠲ඪ΍଎౓,Ճ଎౓͸஋ͷεέʔϧ͕ҟͳΔͨΊ, ಉ͡ωοτϫʔΫͰͷֶश͕ࠔ೉Ͱ͋Δ. ͦ͜Ͱ, ֤Ϟʔμϧ͕࣋ͭྗֶతಛੑ΍஋ͷεέʔϧͷҧ͍ʹ΋ͱ͍ͮͨ Mechanics-streamߏ଄ΛఏҊ͢Δ. Mechanics-stream ߏ଄͸ 3 ͭͷωοτϫʔΫΛ༻ҙ͠, ಛੑ΍஋ͷεέʔϧ͕͍ۙϞʔμϧΛೖ
ྗͱ͢Δ. ֤ωοτϫʔΫ͸ಠֶཱͯ͠शΛߦ͍, ֤ωοτϫʔΫ͔ΒಘΒΕΔΫϥε֬཰ΛΫϥε͝ͱʹ߹‫͢ܭ‬Δ͜ͱͰ࠷ऴతͳΫϥε֬཰ͱ͢Δ.
関節座標
骨情報
関節速度
骨速度
関節加速度
骨加速度

Coordinate network

Velocity network

Acceleration networkਤ 3: Mechanics-stream ߏ଄

3.2 Spatial Temporal Attention Graph

4. ධՁ࣮‫ݧ‬

EAttention node M (XFout ) ͸ؔઅ × ϑϨʔϜͷ 2 ࣍‫ݩ‬ϚοϓͰ͋Γ, ಈ࡞ಛ༗ͷؔઅͷॏཁ౓ΛϑϨʔϜ͝ͱʹද‫͖Ͱݱ‬Δ. ֫ಘͨ͠ Attention node ͸ Attention ‫ߏػ‬EΛ༻͍ͯ Feature extractor ͔Βͷಛ௃Ϛοϓ XFout ΁൓ө͠, Perception branch ͷೖྗͱ͢Δ. ࣜ (1) ʹ Attention‫ߏػ‬Λࣔ͢.
EFEX′out = M (XF(1)out ) · XoutఏҊख๏ͷ༗ޮੑΛ֬ೝ͢ΔͨΊʹධՁ࣮‫ݧ‬Λߦ͏. σʔληοτʹ͸ NTU-RGB+D [2] ͱ NTU-RGB+D120 [3]Λ༻͍Δ. NTU-RGB+D ͸ؔઅ਺ 25 ͷ 3 ࣍‫࠲ݩ‬ඪ (X, Y, Z)Λ࣋ͭಈ࡞ೝࣝ༻ͷσʔληοτͰ͋Γ, ಈ࡞Ϋϥε਺͸60 Ͱ͋Δ. ධՁํ๏͸, 40 ໊ͷඃ‫ऀݧ‬ͷσʔλΛֶशͱ‫Ͱূݕ‬෼͚Δ Cross subject (x-sub) ͱ, 3 ํ޲͔ΒࡱӨͨ͠σʔλΛֶशͱ‫ʹূݕ‬෼͚Δ Cross view (x-view) ͕͋(a) Throw(b) Kick(c) Drinkਤ 4: Attention graph ͷՄࢹԽ: ؔઅͷ৭͸ Attention node ʹ΋ͱ͍͓ͮͯΓ, ੺͍΄Ͳॏཁ౓͕ߴ͘, ੨͍΄Ͳॏཁ౓͕௿͍. ੺͍ઢ͸ Attention edge Λ͓ࣔͯ͠ΓॏΈͷେ্͖͍Ґ 30 ຊͷΈΛඳը͍ͯ͠Δ.
ද 3: Ϟʔμϧͷ૊Έ߹Θͤͷҧ͍ʹΑΔೝࣝਫ਼౓ [%]: ୯ҰͷωοτϫʔΫͷೝࣝਫ਼౓ (Ind. net) ͱ stream ߏ଄Λߏஙͨ͠ͱ͖ͷೝࣝਫ਼౓ (X-stream). ධՁ͸ NTU-RGB+D ͷ Cross subject.
Input data
Joint (coordinate, velocity)
Bone (coordinate, velocity)
Joint (coordinate, velocity, acceleration)
Bone (coordinate, velocity, acceleration)
Coordinate (joint, bone)
Velocity (joint, bone)
Coordinate (joint, bone)
Velocity (joint, bone)
Acceleration (joint, bone)

w/o Attention
Ind. net
X-stream
86.0
87.1
86.1
86.7
87.8
86.7
87.5
89.3
85.6
87.5
85.6
89.1
74.6

ද 1: ैདྷख๏ͱͷೝࣝਫ਼౓ͷൺֱ [%]NTU-RGB+DMethods
x-sub x-view
ST-GCN
81.5
88.3
AS-GCN
86.8
94.2
2s-AGCN
88.5
95.1
AGC-SLTM
89.2
95.0
DGNN
89.9
96.1
STA-GCN
90.1
95.8

NTU-RGB+D120
Methods
x-sub x-setup
2s-ALSTM
61.2
63.3
BPEM
64.6
66.9
SkelMotion
67.7
66.9
TSRJI
67.9
62.8
ST-GCN
72.8
75.4
STA-GCN
83.9
86.5

Δ. NTU-RGB+D120 ͸ NTU-RGB+D ʹ 60 ͷಈ࡞ΫϥεΛ௥Ճͨ͠େ‫ن‬໛ͳσʔληοτͰ͋Δ. ධՁํ๏͸,
Cross subject ͱ, Χϝϥͷߴ͞΍ඃ‫ͱऀݧ‬ͷ‫ʹ཭ڑ‬ΑׂͬͯΓ౰ͯΒΕͨ ID Λ༻͍ͯσʔλΛֶशͱ‫ʹূݕ‬෼͚Δ
Cross setup (x-setup) ͕͋Δ.

4.1 ैདྷख๏ͱͷೝࣝਫ਼౓ͷൺֱද 1 ʹ, ैདྷख๏ͱͷೝࣝਫ਼౓ͷൺֱ݁ՌΛࣔ͢. ͲͪΒͷσʔληοτʹ͓͍ͯ΋ఏҊख๏ͷೝࣝਫ਼౓͸, ैདྷख๏ͱൺֱͯ͠ೝࣝਫ਼౓ͷ޲্, ·ͨ͸ಉ౳ͷೝࣝਫ਼౓Λ
ୡ੒ͨ͠. STA-GCN ͸, ಈ࡞ಛ༗ͷಛ௃Λ‫ڧ‬ௐ͢Δ͜ͱͰྨࣅͨ͠ಈ࡞Ͱ΋ҟͳΔಛ௃Λ֫ಘͰ͖ΔͨΊ, ೝࣝਫ਼౓ͷ޲্ʹߩ‫͍͑ͱͨ͠ݙ‬Δ.

4.2 Attention graph ͷՄࢹԽSTA-GCN ʹΑΓ֫ಘͨ͠ Attention graph Λਤ 4 ʹࣔ͢. ౤͛Δಈ࡞ (ਤ 4(a)) ͸, Τοδ͕ӈ࿹ʹूத͍ͯ͠Δ.
ӈ࿹ͷॏཁ౓͸ಈ࡞தʹঃʑʹߴ͘ͳΓ, ౤͛ऴΘΔͱॏ
ཁ౓͕௿͘ͳΔ. ऽΔಈ࡞ (ਤ 4(b)) ͸, ऽΓग़͢‫ʹ٭‬ରͯ͠Τοδ͕ूத͢Δ. ͜ͷ͜ͱ͔Β, ಈ࡞ಛ༗ͷ Attention
edge Λ֫ಘͨ͠ͱ͍͑Δ. ҿΉಈ࡞ (ਤ 4(c)) ͸౤͛Δͱಉ༷ʹӈ࿹Λॏཁࢹ͍ͯ͠Δ. ͔͠͠ͳ͕Β, ҿΉಈ࡞ͷ‫٭‬ͷॏཁ౓͸Ұ؏ͯ͠௿͍ͷʹର͠, ౤͛Δಈ࡞͸ମॏҠಈΛߦ͏ͨΊ‫٭‬ͷॏཁ౓͕ߴ͘ͳΔ͜ͱ͔Β, ಈ࡞ಛ༗ͷ
Attention node Λ֫ಘͨ͠ͱ͍͑Δ.

4.3 Ablation study
Attention graph ͷैདྷख๏΁ͷద༻: ද 2 ʹ, Attention graph Λैདྷख๏΁ద༻ͨ͠ͱ͖ͷೝࣝਫ਼౓Λࣔ͢.
ද 2 ͔Β, Attention edge, ͓Αͼ Attention node ͷΈΛద༻ͨ͠΄΅શͯͷ৔߹Ͱೝࣝਫ਼౓ͷ޲্Λ֬ೝͰ͖Δ. ·ͨ, શͯͷैདྷख๏ʹ͓͍ͯ Attention edge ͱ Attention
node ͷ྆ํΛద༻ͨ͠৔߹ʹ࠷΋ೝࣝਫ਼౓͕ߴ͘ͳΔ. ͜ͷ݁Ռ͔Β, Attention graph ͸ೝࣝਫ਼౓ͷ޲্ʹߩ‫͠ݙ‬͓ͯΓ, ྆ํΛಉ࣌ʹద༻ͨ͠৔߹ʹ࠷΋ޮՌతͰ͋Δ.

w/ Attention
Ind. net
X-stream
86.2
87.2
86.3
85.7
86.7
85.8
88.6
90.1
87.0
88.6
87.0
89.4
77.2

ද 2: Attention graph ͷैདྷख๏΁ͷద༻݁Ռ [%]:ධՁ͸ NTU-RGB+D ͷ Cross subject.
Attentionedge×!
×!

Attention
node××!
!

ST-GCN

AS-GCN

2s-AGCN

81.5
84.1
82.8
84.8

86.8
86.9
86.8
87.0

88.5
89.2
89.1
89.3

Ϟʔμϧͷ૊Έ߹Θͤͷҧ͍ʹΑΔೝࣝਫ਼౓: ද 3 ʹ, ೖྗϞʔμϧͷҧ͍ʹΑΔೝࣝਫ਼౓Λࣔ͢. ද 3 ͔Β, ࠲ඪͱ଎౓Λಉ͡ωοτϫʔΫͰֶश͢ΔΑΓ, ผʑͷωοτϫʔΫͰֶशͨ͠ํ͕ೝࣝਫ਼౓͕ߴ͘ͳΔ. ͜ͷ͜ͱ͔Β,
Mechanics-stream ߏ଄ͷ༗ޮੑΛ֬ೝͰ͖Δ. ͔͠͠ͳ͕Β, Ճ଎౓Λೖྗ͢Δ৔߹Ͱ͸ೝࣝਫ਼౓͕௿Լͨ͠. Ճ଎౓͸஋ͷεέʔϧ͕খ͘͞, ೝࣝͷͨΊͷे෼ͳಛ௃͕
‫·ؚ‬Ε͍ͯͳ͍͜ͱ͕ߟ͑ΒΕΔ.

5. ͓ΘΓʹຊ‫Ͱڀݚ‬͸, ؔઅͷॏཁ౓ͱؔ܎ੑΛߟྀͯ͠ಈ࡞ೝࣝΛߦ͏ Spatial Temporal Attention GCN ΛఏҊͨ͠. ·ͨ,ϚϧνϞʔμϧֶशʹ͓͚ΔϞʔμϧͷಛੑͷҧ͍ʹ΋ͱ͍ͮͯωοτϫʔΫߏங͢Δ Mechanics-stream ߏ଄ΛఏҊͨ͠. ධՁ࣮‫Ͱݧ‬͸, ఏҊख๏͕ैདྷख๏Λ௒͑Δೝࣝਫ਼౓Λୡ੒͠༗ޮੑΛ֬ೝͨ͠. ·ͨ, ಈ࡞ಛ༗ͷ Attention
graph ͕֫ಘ͢Δ͜ͱΛ֬ೝͨ͠. ࠓ‫ޙ‬ͷ՝୊ͱͯ͠, ಈ࡞ೝࣝҎ֎ͷλεΫ΁ͷԠ༻ͳͲ͕͋͛ΒΕΔ.

ࢀߟจ‫ݙ‬[1] S. Yan, et al., “Spatial Temporal Graph Convolutional Networks for Skeleton-Based Action Recognition”,AAAI, 2018.
[2] S. Amir, et al., “NTU RGB+D: A Large Scale Dataset
for 3D Human Activity Analysis”, CVPR, 2016.
[3] L. Jun, et al., “NTU RGB+D 120: A Large-Scale Benchmark for 3D Human Activity Understanding”, TPAMI,2019.

‫੷ۀڀݚ‬[1] K. Shiraki, et al., “Spatial Temporal Attention Graph
Convolutional Networks with Mechanics-Stream for
Skeleton-based Action Recognition”, ACCV, 2020.
(ଞ 3 ݅)