機械知覚&ロボティクスグループ
中部大学 工学研究科 情報工学専攻

2020年度 山下研究室 修士論文発表 アブストラクト
Deep Learning Action Recognition Graph Convolutional Networks
時空間のアテンション情報を考慮したGraph Convolutional Networksによる動作認識に関する研究

白木 克俊
Deep Learning Object Detection Point Clouds

Study of 3D Object Detection with Normal-map on Point Clouds繆 継樹
Deep Learning Reinforcement Learning Visual Explanation

Attention機構による視覚的説明を可能とした深層強化学習に関する研究
板谷 英典
Deep Learning Change Detection Semantic Segmentation

セマンティックセグメンテーションによる超高解像度画像からの変化点検出に関する研究
筒井 駿吾
Deep Learning Saliency Prediction

異なる解像度間の整合性を考慮した顕著性予測モデルに関する研究
瀬尾 俊貴
Deep Learning Self-driving Gaze

視線情報を活用した一貫学習ベースの自動運転制御に関する研究
森 啓介

URL: http://mprg.jpۭ࣌ؒͷΞςϯγϣϯ৘ใΛߟྀͨ͠ Graph Convolutional Networks ʹΑΔಈ࡞ೝࣝʹؔ͢Δ‫ڀݚ‬ࢦಋ‫ڭ‬तɿࢁԼོٛ

TP19008 ന໦ࠀढ़2. ࠎ֨σʔλ͔Βͷ GCN Λ༻͍ͨಈ࡞ೝࣝ
ࠎ֨σʔλ͔Βͷಈ࡞ೝࣝʹ, GCN Λ༻͍ͨख๏Ͱ͋Δ
Spatial Temporal GCN (ST-GCN) [1] ͕͋Δ. ST-GCN
͸, ಉҰϑϨʔϜ಺ͷؔઅΛ݁Ϳۭؒάϥϑͱ, ྡ઀ϑϨʔϜͷಉҰؔઅΛ݁Ϳ࣌ؒάϥϑͷ 2 ͭͷάϥϑߏ଄ (ਤ 1)Ͱࠎ֨σʔλΛද‫͢ݱ‬Δ͜ͱʹΑΓߴ͍ೝࣝਫ਼౓Λୡ੒͠
ͨ. ͔͠͠ͳ͕Β, ST-GCN ͸, ۭؒάϥϑͷ઀ଓؔ܎ ESΛࠎ֨ύλʔϯ H Ͱఆ͍ٛͯ͠ΔͨΊ, ಈ࡞ಛ༗ͷؔઅؒͷؔ܎ੑΛߟྀͰ͖ͳ͍. ·ͨ, ಈ࡞΍࣌ؒ͝ͱʹมԽ͢Δؔઅͷॏཁ౓΋ද‫͏͍ͱ͍ͳ͖Ͱݱ‬໰୊͕͋Δ.

時間グラフ： EF = {vti , v(t 1)i }<latexit sha1_base64="WO6MoIs0XDy6Os0H0DA6rik5Ljk=">AAACgXichVFNS+RAEH1G1x2jq6NeBD3IDoqCO1REUFwEURSPfo0KjoQk27qNmSQkPQOzYS573D/gwZOCqHhy/8Je/AMe/AmLRwUvHqzJBBYV1wrpev26XvXrbjtwZaSIbpq05pYPrR8zbXp7x6fOrmx3z0bkl0NHFBzf9cMt24qEKz1RUFK5YisIhVWyXbFp78/X1zcrIoyk762raiB2StaeJ3elYymmzOzAgrk4oxdjvWLGStbGOI2oL8aorOnFmpnNUZ6SGHwNjBTkkMaynz1FEd/gw0EZJQh4UIxdWIj424YBQsDcDmLmQkYyWReoQWdtmasEV1jM7vO4x7PtlPV4Xu8ZJWqHd3H5D1k5iCG6pnO6oyu6oL/0+GavOOlR91LlbDe0IjC7fvWtPbyrKnFW+P5P9V/PCruYSrxK9h4kTP0UTkNf+XFwtza9OhQP0zHdsv8juqE/fAKvcu+crIjVQ+j8AMbL634NNsbzBuWNlYnc7Fz6FBn04zNG+L4nMYslLKPA+/7EGS7xW2vWRjXSxhulWlOq6cWz0L4+AR5fkuQ=</latexit>#：骨格パターン!：フレーム"：ノード数ਤ 1: ST-GCN ʹ͓͚Δࠎ֨σʔλͷάϥϑද‫ݱ‬

3. ఏҊख๏
ؔઅͷॏཁ౓ͱؔ܎ੑΛߟྀͯ͠ಈ࡞ೝࣝΛߦ͏ Spatial Temporal Attention GCN (STA-GCN) ΛఏҊ͢Δ.
STA-GCN ͸ؔઅͷॏཁ౓Ͱ͋Δ Attention node ͱ, ؔઅؒͷؔ܎ੑͰ͋Δ Attention edge Λ֫ಘ͢Δ. ֫ಘͨ͠ Attention Λߟྀͭͭ͠ೝࣝΛߦ͏. ·ͨ, ϚϧνϞʔμϧֶशʹ͓͍ͯ Mechanics-stream ߏ଄ΛఏҊ͢Δ. Mechanicsstream ߏ଄͸, ֤Ϟʔμϧ͕࣋ͭྗֶతಛੑ΍஋ͷεέʔϧͷҧ͍ʹ΋ͱ͍ͮͯωοτϫʔΫΛߏங͢Δ.

3.1 STA-GCN ͷωοτϫʔΫߏ଄
ਤ 2 ʹ STA-GCN ͷωοτϫʔΫߏ଄Λࣔ͢. STAGCN ͸, Feature extractor, Attention branch, Perception branch ͷ 3 ͭͷϞδϡʔϧͰߏ੒͢Δ. ֤Ϟδϡʔϧʹ‫·ؚ‬ΕΔ STGC-block Ͱ͸, ۭؒάϥϑͱ࣌ؒάϥϑͷάϥϑ৞ΈࠐΈॲཧΛߦ͏. Feature extractor ʹ͸ 2 ͭͷϞʔμϧΛೖྗ͠, ਓؒͷࠎ֨ύλʔϯ Askeleton Λۭؒάϥϑͱ͢Δෳ਺ͷ STGC-block ʹΑΓ֤Ϟʔμϧͷಛ௃ϚοϓΛ֫ಘ͠, ݁߹͢Δ. Attention branch ͸, ؔઅͷॏཁ౓Λද͢ Attention node ͱ, ؔઅؒͷॏཁͳؔ܎ੑΛද͢ Attention edge Λಛ௃Ϛοϓ͔Β֫ಘ͢Δ. Perception
branch ͸, Attention node ͱ Attention edge Λߟྀͭͭ͠, ࠷ऴతͳΫϥε֬཰Λग़ྗ͢Δ. ֶश͸, 2 ͭͷϒϥϯν͔Βͷग़ྗʹରͯ͠ΫϩεΤϯτϩϐʔ‫Ͱࠩޡ‬ଛࣦΛ‫ٻ‬Ί, ଛࣦͷ࿨ΛωοτϫʔΫͷֶश‫͍༻ͯ͠ͱࠩޡ‬Δ.

Batch
Normalization

FC

Softmax

FC

Softmax

1 1 conv.
1 1 conv.

GAP

Tanh

Batch
Normalization

ReLU

1 1 conv.

GAP

Latt

Lper

<latexit sha1_base64="rzLIR/W4Zap6hMNhKgk0nYv5i30=">AAACanichVG7SgNBFD1ZXzE+EmOjpBGjYhXuiqBYCTYWFiYxRlAJu+uoSza7y+4kEEN+wM5K0EpBRPwMG3/AIp8g2inYWHizWRAV9Q4zc+bMPXfOzOiuZfqSqBVRurp7evui/bGBwaHheGIkuek7Vc8QBcOxHG9L13xhmbYoSFNaYsv1hFbRLVHUyyvt/WJNeL7p2Buy7ordinZgm/umoUmmimulhiZls5RIU4aCmPgJ1BCkEca6k7jGDvbgwEAVFQjYkIwtaPC5bUMFwWVuFw3mPEZmsC/QRIy1Vc4SnKExW+bxgFfbIWvzul3TD9QGn2Jx91g5gWl6oBt6oXu6pUd6/7VWI6jR9lLnWe9ohVuKH4/l3/5VVXiWOPxU/elZYh+LgVeTvbsB076F0dHXjk5f8ku56cYMXdIT+7+gFt3xDezaq3GVFblzxPgD1O/P/RNszmVUyqjZ+fRyLvyKKFKYxCy/9wKWsYp1FAJ3JzjDeeRZSSrjSqqTqkRCzSi+hDL1AcvijLg=</latexit>

Askeleton

STGC-Block 5<latexit sha1_base64="aYu0bwSDva65Ct14jn/gNrFJXWM=">AAACd3ichVHLSsNAFD2N7/qquhFcWCyKq3IjguJKcePStlYFHyWJUw1Nk5BMixr6A/6AC0FQkCp+hht/wEU/QVwqiODC2zQgKuodZubMmXvunJnRXcv0JVEjprS1d3R2dffEe/v6BwYTQ8PrvlPxDJE3HMvxNnXNF5Zpi7w0pSU2XU9oZd0SG3ppubm/URWebzr2mjxyxU5Z27fNomlokqlCYjjY1ovJpdpu4JeEJaRj1wqJFKUpjORPoEYghShWnUQd29iDAwMVlCFgQzK2oMHntgUVBJe5HQTMeYzMcF+ghjhrK5wlOENjtsTjPq+2ItbmdbOmH6oNPsXi7rEyiUl6oBt6pnu6pUd6/7VWENZoejniWW9phVsYPBnNvf6rKvMscfCp+tOzRBHzoVeTvbsh07yF0dJXj0+fcwvZyWCKLumJ/V9Qg+74Bnb1xbjKiOwZ4vwB6vfn/gnWZ9IqpdXMbGoxG31FN8YwgWl+7zksYgWryPO5hzhHHdexN2VcmVKmW6lKLNKM4Eso6ge6I5Gj</latexit>

Perception branch

X%&
"#$

STGC-Block 5X/"#$

Askeleton

STGC-Block 5<latexit sha1_base64="aYu0bwSDva65Ct14jn/gNrFJXWM=">AAACd3ichVHLSsNAFD2N7/qquhFcWCyKq3IjguJKcePStlYFHyWJUw1Nk5BMixr6A/6AC0FQkCp+hht/wEU/QVwqiODC2zQgKuodZubMmXvunJnRXcv0JVEjprS1d3R2dffEe/v6BwYTQ8PrvlPxDJE3HMvxNnXNF5Zpi7w0pSU2XU9oZd0SG3ppubm/URWebzr2mjxyxU5Z27fNomlokqlCYjjY1ovJpdpu4JeEJaRj1wqJFKUpjORPoEYghShWnUQd29iDAwMVlCFgQzK2oMHntgUVBJe5HQTMeYzMcF+ghjhrK5wlOENjtsTjPq+2ItbmdbOmH6oNPsXi7rEyiUl6oBt6pnu6pUd6/7VWENZoejniWW9phVsYPBnNvf6rKvMscfCp+tOzRBHzoVeTvbsh07yF0dJXj0+fcwvZyWCKLumJ/V9Qg+74Bnb1xbjKiOwZ4vwB6vfn/gnWZ9IqpdXMbGoxG31FN8YwgWl+7zksYgWryPO5hzhHHdexN2VcmVKmW6lKLNKM4Eso6ge6I5Gj</latexit><latexit sha1_base64="z3Ou31v8Q/t/Y9rkeF3QiqHVQJk=">AAACanichVG7SgNBFD1ZXzE+EmOjpBGjYhXuiqBYBWwsLPIwRlAJu+uoSza7y+4mEEN+wM5KMJWCiPgZNv6ART5BtItgY+HNZkFU1DvMzJkz99w5M6Pahu56RO2Q1Nc/MDgUHo6MjI6NR2MT8W3XqjqaKGiWYTk7quIKQzdFwdM9Q+zYjlAqqiGKanm9u1+sCcfVLXPLq9tiv6IcmfqhrikeU8XNUsMWTrMUS1KK/Jj5CeQAJBFExordYA8HsKChigoETHiMDShwue1CBsFmbh8N5hxGur8v0ESEtVXOEpyhMFvm8YhXuwFr8rpb0/XVGp9icHdYOYN5eqRb6tAD3dETvf9aq+HX6Hqp86z2tMIuRU+n8m//qio8ezj+VP3p2cMhVn2vOnu3faZ7C62nr52cd/JrufnGAl3RM/u/pDbd8w3M2qt2nRW5FiL8AfL35/4JtpdSMqXk7HIynQu+IowEZrHI772CNDaQQcF3d4YLtEIvUlyalhK9VCkUaCbxJaS5D8fvjLY=</latexit>

Aattention , Askeleton<latexit sha1_base64="cBh/6QephkG1NQXqYsZzQlLsuMQ=">AAACjHicSyrIySwuMTC4ycjEzMLKxs7BycXNw8vHLyAoFFacX1qUnBqanJ+TXxSRlFicmpOZlxpaklmSkxpRUJSamJuUkxqelO0Mkg8vSy0qzszPCympLEiNzU1Mz8tMy0xOLAEKxQuoV8ckpSk41sZVJ5aUpOaBBGt1FOCCxdmpOaklQLF4AWUDPQMwUMBkGEIZygxQEJAvsJwhhiGFIZ8hmaGUIZchlSGPoQTIzmFIZCgGwmgGQwYDhgKgWCxDNVCsCMjKBMunMtQycAH1lgJVpQJVJAJFs4FkOpAXDRXNA/JBZhaDdScDbckB4iKgTgUGVYOrBisNPhucMFht8NLgD06zqsFmgNxSCaSTIHpTC+L5uySCvxPUlQukSxgyELrwurmEIY3BAuzWTKDbC8AiIF8kQ/SXVU3/HGwVpFqtZrDI4DXQ/QsNbhocBvogr+xL8tLA1KDZDFzACDBED25MRpiRnqGBnmGgibJDEDQqOBikGZQYNIDhbc7gwODBEMAQCrS3l2Ezwx6GvUx8TCZM1ky2EKVMjFA9wgwogMkNAHa1mWs=</latexit>

ਤ 2: Spatial Temporal Attention GCN

Attention edge ͸, ಈ࡞͝ͱͷؔઅؒͷॏཁͳؔ܎ੑΛද͢ྡ઀ߦྻͰ͋Δ. ֫ಘͨ͠ Attention edge ͸ Perception
branch ʹ༩͑Δ. Perception branch Ͱ͸, Attention edge
Aattention ͱਓؒͷࠎ֨ύλʔϯ Askeleton ʹΑΔάϥϑ
৞ΈࠐΈॲཧΛߦ͏. Perception branch ʹ͓͚Δೖྗಛ௃ Xin ʹର͢Δۭؒάϥϑ৞ΈࠐΈॲཧΛࣜ (2) ʹࣔ͢.
Mskel , Wskel , Watt ͸ॏΈߦྻͰ͋Δ. Q ͸ hop ਺Ͱ͋Γ Q ‫཭ݸ‬Εͨؔઅ·ͰΛ݁Ϳ. K ͸ಈ࡞͝ͱʹੜ੒͢Δ
Attention edge ͷ਺Ͱ͋Δ. ຊ‫Ͱڀݚ‬͸ Q = 3, K = 4 ͱ
͢Δ.
Q!
Xout =
Mskel◦ Âskel
Xin Wqskel
q
q
q

+

<latexit sha1_base64="045F/2C3Ksius+g4s7VfI3ujK8o=">AAACjHicSyrIySwuMTC4ycjEzMLKxs7BycXNw8vHLyAoFFacX1qUnBqanJ+TXxSRlFicmpOZlxpaklmSkxpRUJSamJuUkxqelO0Mkg8vSy0qzszPCympLEiNzU1Mz8tMy0xOLAEKxQuoh9nGVHOVxVeXZNbWlNga6sSk5JcU64ToKGTCOX5cMbXxAsoGegZgoIDJMIQylBmgICBfYDlDDEMKQz5DMkMpQy5DKkMeQwmQncOQyFAMhNEMhgwGDAVAsViGaqBYEZCVCZZPZahl4ALqLQWqSgWqSASKZgPJdCAvGiqaB+SDzCwG604G2pIDxEVAnQoMqgZXDVYafDY4YbDa4KXBH5xmVYPNALmlEkgnQfSmFsTzd0kEfyeoKxdIlzBkIHThdXMJQxqDBditmUC3F4BFQL5Ihugvq5r+OdgqSLVazWCRwWug+xca3DQ4DPRBXtmX5KWBqUGzGbiAEWCIHtyYjDAjPUMDPcNAE2UHJ2hUcDBIMygxaADD25zBgcGDIYAhFGhvL8Nmhj0Me5n4mEyYrJlsIUqZGKF6hBlQAJMbAPxCly4=</latexit><latexit sha1_base64="Agw0HGDAh3iEqbKTurGdvpX284A=">AAACiHicSyrIySwuMTC4ycjEzMLKxs7BycXNw8vHLyAoFFacX1qUnBqanJ+TXxSRlFicmpOZlxpaklmSkxpRUJSamJuUkxqelO0Mkg8vSy0qzszPCympLEiNzU1Mz8tMy0xOLAEKxQsou8YH23LFVHOVxVeXZNbqgKis2hqNTJ0szZjMPAUPrphaoCoDPQMwUMBkGEIZygxQEJAvsJwhhiGFIZ8hmaGUIZchlSGPoQTIzmFIZCgGwmgGQwYDhgKgWCxDNVCsCMjKBMunMtQycAH1lgJVpQJVJAJFs4FkOpAXDRXNA/JBZhaDdScDbckB4iKgTgUGVYOrBisNPhucMFht8NLgD06zqsFmgNxSCaSTIHpTC+L5uySCvxPUlQukSxgyELrwurmEIY3BAuzWTKDbC8AiIF8kQ/SXVU3/HGwVpFqtZrDI4DXQ/QsNbhocBvogr+xL8tLA1KDZDFzACDBED25MRpiRnqGBnmGgibKDEzQqOBikGZQYNIDhbc7gwODBEMAQCrS3k2E9ww6GnUxcTAZM5kyWEKVMjFA9wgwogMkJAATplfA=</latexit>

STGC-Block 5

Feature extractor

ノード：V = {vti |t = 1, . . . , T, i = 1, . . . , N }空間グラフ： ES = {vti , vtj |(i, j) 2 H}

Batch
Normalization

Attention node '(X %&"#$ )Sigmoidࠎ֨σʔλ͸ਓؒͷಈ͖Λ௚઀ଊ͑Δ͜ͱ͕Ͱ͖Δ͜ͱ΍, ೝࣝ࣌ʹ͓͚Δ‫ڥ؀‬΍ࢹ఺ͷมԽʹରͯ͠‫ͳ݈ؤ‬ར఺͕
͋Δ. ࠎ֨σʔλΛ༻͍ͨ Graph Convolutional Networks(GCN) ʹΑΔैདྷͷಈ࡞ೝࣝͰ͸, ਓؒͷࠎ֨ύλʔϯͰάϥϑߏ଄Λࣄલʹఆٛ͢ΔͨΊ, ಈ࡞ಛ༗ͷؔઅؒͷؔ܎ੑΛߟྀͰ͖ͳ͍. ·ͨ, ೝࣝʹ͓͚Δؔઅͷॏཁ౓΋ಈ࡞͝ͱʹҟͳΔ͜ͱ͕༧૝͞ΕΔ. ຊ‫Ͱڀݚ‬͸, ؔઅͷॏཁ౓ͱؔ܎ੑΛߟྀͯ͠ಈ࡞ೝࣝΛߦ͏ Spatial Temporal
Attention Graph Convolutional Networks (STA-GCN)
ΛఏҊ͢Δ. STA-GCN ͸, ϑϨʔϜ͝ͱͷؔઅͷॏཁ౓ͱ, ಈ࡞͝ͱͷؔઅؒͷ઀ଓؔ܎Λ֫ಘ͢Δ. ॏཁ౓Λಛ௃ϚοϓʹॏΈ෇͚͠, ઀ଓؔ܎Λ༻͍ͯάϥϑ৞ΈࠐΈॲཧΛߦ͏͜ͱͰ, ॏཁ౓ͱؔ܎ੑΛߟྀͭͭ͠ೝࣝΛߦ
͏. ·ͨ, ϚϧνϞʔμϧֶशʹ͓͍ͯ Mechanics-streamߏ଄Λಋೖ͠, ߴਫ਼౓ԽΛ࣮‫͢ݱ‬Δ.

Attention branch

Upsampling

Attention edge A+$$,-$."-GAP

1. ͸͡Ίʹ

K
!

(2)

att
Âatt
k Xin Wkk

Attention node ͱ Attention edge ʹΑͬͯ࣌ؒత, ۭؒతͳಛ௃Λ‫ڧ‬ௐͰ͖, 2 ͭͷ Attention Λ૊Έ߹Θͤͨάϥϑද‫ݱ‬Λ Spatial Temporal Attention Graph (Attention
graph) ͱ‫Ϳݺ‬.

3.3 Mechanics-stream ߏ଄
Mechanics-stream ߏ଄ (ਤ 3) ͸, ϚϧνϞʔμϧֶशʹ͓͚ΔωοτϫʔΫߏ଄Ͱ͋Δ. ຊ‫Ͱڀݚ‬͸, ؔઅ࠲ඪ͔Βؔઅ଎౓, ؔઅՃ଎౓, ࠎ৘ใ, ࠎ଎౓, ࠎՃ଎౓Λࢉग़͠ωοτϫʔΫͷೖྗͱ͢Δ. ࠎ৘ใ͸, ؔઅؒͷ‫཭ڑ‬Ͱ͋Γؔઅؒͷํ޲΋ද‫͢ݱ‬Δ. ࠲ඪ΍ࠎ৘ใ͸ۭؒతͳҐஔͷ৘ใ, ଎౓΍Ճ଎౓͸࣌ؒతͳҠಈྔͷ৘ใͰ͋Γ,֤Ϟʔμϧ͕࣋ͭಛੑʹ͸ҧ͍͕͋Δ. ·ͨ, ࠲ඪ΍଎౓,Ճ଎౓͸஋ͷεέʔϧ͕ҟͳΔͨΊ, ಉ͡ωοτϫʔΫͰͷֶश͕ࠔ೉Ͱ͋Δ. ͦ͜Ͱ, ֤Ϟʔμϧ͕࣋ͭྗֶతಛ
ੑ΍஋ͷεέʔϧͷҧ͍ʹ΋ͱ͍ͮͨ Mechanics-streamߏ଄ΛఏҊ͢Δ. Mechanics-stream ߏ଄͸ 3 ͭͷωοτϫʔΫΛ༻ҙ͠, ಛੑ΍஋ͷεέʔϧ͕͍ۙϞʔμϧΛೖ
ྗͱ͢Δ. ֤ωοτϫʔΫ͸ಠֶཱͯ͠शΛߦ͍, ֤ωοτϫʔΫ͔ΒಘΒΕΔΫϥε֬཰ΛΫϥε͝ͱʹ߹‫͢ܭ‬Δ͜ͱͰ࠷ऴతͳΫϥε֬཰ͱ͢Δ.
関節座標
骨情報
関節速度
骨速度
関節加速度
骨加速度

Coordinate network

Velocity network

Acceleration networkਤ 3: Mechanics-stream ߏ଄

3.2 Spatial Temporal Attention Graph

4. ධՁ࣮‫ݧ‬E
Attention node M (XF
out ) ͸ؔઅ × ϑϨʔϜͷ 2 ࣍‫ݩ‬ϚοϓͰ͋Γ, ಈ࡞ಛ༗ͷؔઅͷॏཁ౓ΛϑϨʔϜ͝ͱʹද‫͖Ͱݱ‬Δ. ֫ಘͨ͠ Attention node ͸ Attention ‫ߏػ‬EΛ༻͍ͯ Feature extractor ͔Βͷಛ௃Ϛοϓ XF
out ΁൓ө͠, Perception branch ͷೖྗͱ͢Δ. ࣜ (1) ʹ Attention‫ߏػ‬Λࣔ͢.
E
FE
X!out = M (XF(1)
out ) · XoutఏҊख๏ͷ༗ޮੑΛ֬ೝ͢ΔͨΊʹධՁ࣮‫ݧ‬Λߦ͏. σʔληοτʹ͸ NTU-RGB+D [2] ͱ NTU-RGB+D120 [3]Λ༻͍Δ. NTU-RGB+D ͸ؔઅ਺ 25 ͷ 3 ࣍‫࠲ݩ‬ඪ (X, Y, Z)Λ࣋ͭಈ࡞ೝࣝ༻ͷσʔληοτͰ͋Γ, ಈ࡞Ϋϥε਺͸60 Ͱ͋Δ. ධՁํ๏͸, 40 ໊ͷඃ‫ऀݧ‬ͷσʔλΛֶशͱ‫Ͱূݕ‬෼͚Δ Cross subject (x-sub) ͱ, 3 ํ޲͔ΒࡱӨͨ͠σʔλΛֶशͱ‫ʹূݕ‬෼͚Δ Cross view (x-view) ͕͋(a) Throw(b) Kick(c) Drinkਤ 4: Attention graph ͷՄࢹԽ: ؔઅͷ৭͸ Attention node ʹ΋ͱ͍͓ͮͯΓ, ੺͍΄Ͳॏཁ౓͕ߴ͘, ੨͍΄Ͳॏཁ౓͕௿͍. ੺͍ઢ͸ Attention edge Λ͓ࣔͯ͠ΓॏΈͷେ্͖͍Ґ 30 ຊͷΈΛඳը͍ͯ͠Δ.
ද 3: Ϟʔμϧͷ૊Έ߹Θͤͷҧ͍ʹΑΔೝࣝਫ਼౓ [%]: ୯ҰͷωοτϫʔΫͷೝࣝਫ਼౓ (Ind. net) ͱ stream ߏ଄Λߏஙͨ͠ͱ͖ͷೝࣝਫ਼౓ (X-stream). ධՁ͸ NTU-RGB+D ͷ Cross subject.
Input data
Joint (coordinate, velocity)
Bone (coordinate, velocity)
Joint (coordinate, velocity, acceleration)
Bone (coordinate, velocity, acceleration)
Coordinate (joint, bone)
Velocity (joint, bone)
Coordinate (joint, bone)
Velocity (joint, bone)
Acceleration (joint, bone)

w/o Attention
Ind. net
X-stream
86.0
87.1
86.1
86.7
87.8
86.7
87.5
89.3
85.6
87.5
85.6
89.1
74.6

ද 1: ैདྷख๏ͱͷೝࣝਫ਼౓ͷൺֱ [%]NTU-RGB+D
Methods
x-sub x-view
ST-GCN
81.5
88.3
AS-GCN
86.8
94.2
2s-AGCN
88.5
95.1
AGC-SLTM
89.2
95.0
DGNN
89.9
96.1
STA-GCN
90.1
95.8

NTU-RGB+D120
Methods
x-sub x-setup
2s-ALSTM
61.2
63.3
BPEM
64.6
66.9
SkelMotion
67.7
66.9
TSRJI
67.9
62.8
ST-GCN
72.8
75.4
STA-GCN
83.9
86.5

Δ. NTU-RGB+D120 ͸ NTU-RGB+D ʹ 60 ͷಈ࡞ΫϥεΛ௥Ճͨ͠େ‫ن‬໛ͳσʔληοτͰ͋Δ. ධՁํ๏͸,
Cross subject ͱ, Χϝϥͷߴ͞΍ඃ‫ͱऀݧ‬ͷ‫ʹ཭ڑ‬ΑׂͬͯΓ౰ͯΒΕͨ ID Λ༻͍ͯσʔλΛֶशͱ‫ʹূݕ‬෼͚Δ
Cross setup (x-setup) ͕͋Δ.

4.1 ैདྷख๏ͱͷೝࣝਫ਼౓ͷൺֱ
ද 1 ʹ, ैདྷख๏ͱͷೝࣝਫ਼౓ͷൺֱ݁ՌΛࣔ͢. ͲͪΒͷσʔληοτʹ͓͍ͯ΋ఏҊख๏ͷೝࣝਫ਼౓͸, ैདྷख๏ͱൺֱͯ͠ೝࣝਫ਼౓ͷ޲্, ·ͨ͸ಉ౳ͷೝࣝਫ਼౓Λ
ୡ੒ͨ͠. STA-GCN ͸, ಈ࡞ಛ༗ͷಛ௃Λ‫ڧ‬ௐ͢Δ͜ͱ
Ͱྨࣅͨ͠ಈ࡞Ͱ΋ҟͳΔಛ௃Λ֫ಘͰ͖ΔͨΊ, ೝࣝਫ਼౓ͷ޲্ʹߩ‫͍͑ͱͨ͠ݙ‬Δ.

4.2 Attention graph ͷՄࢹԽ
STA-GCN ʹΑΓ֫ಘͨ͠ Attention graph Λਤ 4 ʹࣔ͢. ౤͛Δಈ࡞ (ਤ 4(a)) ͸, Τοδ͕ӈ࿹ʹूத͍ͯ͠Δ.
ӈ࿹ͷॏཁ౓͸ಈ࡞தʹঃʑʹߴ͘ͳΓ, ౤͛ऴΘΔͱॏ
ཁ౓͕௿͘ͳΔ. ऽΔಈ࡞ (ਤ 4(b)) ͸, ऽΓग़͢‫ʹ٭‬ରͯ͠Τοδ͕ूத͢Δ. ͜ͷ͜ͱ͔Β, ಈ࡞ಛ༗ͷ Attention
edge Λ֫ಘͨ͠ͱ͍͑Δ. ҿΉಈ࡞ (ਤ 4(c)) ͸౤͛Δͱಉ༷ʹӈ࿹Λॏཁࢹ͍ͯ͠Δ. ͔͠͠ͳ͕Β, ҿΉಈ࡞ͷ‫٭‬ͷॏཁ౓͸Ұ؏ͯ͠௿͍ͷʹର͠, ౤͛Δಈ࡞͸ମॏҠಈΛߦ͏ͨΊ‫٭‬ͷॏཁ౓͕ߴ͘ͳΔ͜ͱ͔Β, ಈ࡞ಛ༗ͷ
Attention node Λ֫ಘͨ͠ͱ͍͑Δ.

4.3 Ablation study
Attention graph ͷैདྷख๏΁ͷద༻: ද 2 ʹ, Attention graph Λैདྷख๏΁ద༻ͨ͠ͱ͖ͷೝࣝਫ਼౓Λࣔ͢.
ද 2 ͔Β, Attention edge, ͓Αͼ Attention node ͷΈΛద༻ͨ͠΄΅શͯͷ৔߹Ͱೝࣝਫ਼౓ͷ޲্Λ֬ೝͰ͖Δ. ·ͨ, શͯͷैདྷख๏ʹ͓͍ͯ Attention edge ͱ Attention
node ͷ྆ํΛద༻ͨ͠৔߹ʹ࠷΋ೝࣝਫ਼౓͕ߴ͘ͳΔ. ͜ͷ݁Ռ͔Β, Attention graph ͸ೝࣝਫ਼౓ͷ޲্ʹߩ‫͠ݙ‬͓ͯΓ, ྆ํΛಉ࣌ʹద༻ͨ͠৔߹ʹ࠷΋ޮՌతͰ͋Δ.

w/ Attention
Ind. net
X-stream
86.2
87.2
86.3
85.7
86.7
85.8
88.6
90.1
87.0
88.6
87.0
89.4
77.2

ද 2: Attention graph ͷैདྷख๏΁ͷద༻݁Ռ [%]:
ධՁ͸ NTU-RGB+D ͷ Cross subject.
Attention
edge
×
!
×
!

Attention
node
×
×
!
!

ST-GCN

AS-GCN

2s-AGCN

81.5
84.1
82.8
84.8

86.8
86.9
86.8
87.0

88.5
89.2
89.1
89.3

Ϟʔμϧͷ૊Έ߹Θͤͷҧ͍ʹΑΔೝࣝਫ਼౓: ද 3 ʹ, ೖྗϞʔμϧͷҧ͍ʹΑΔೝࣝਫ਼౓Λࣔ͢. ද 3 ͔Β, ࠲ඪͱ଎౓Λಉ͡ωοτϫʔΫͰֶश͢ΔΑΓ, ผʑͷωοτϫʔΫͰֶशͨ͠ํ͕ೝࣝਫ਼౓͕ߴ͘ͳΔ. ͜ͷ͜ͱ͔Β,
Mechanics-stream ߏ଄ͷ༗ޮੑΛ֬ೝͰ͖Δ. ͔͠͠ͳ͕Β, Ճ଎౓Λೖྗ͢Δ৔߹Ͱ͸ೝࣝਫ਼౓͕௿Լͨ͠. Ճ଎౓͸஋ͷεέʔϧ͕খ͘͞, ೝࣝͷͨΊͷे෼ͳಛ௃͕
‫·ؚ‬Ε͍ͯͳ͍͜ͱ͕ߟ͑ΒΕΔ.

5. ͓ΘΓʹ
ຊ‫Ͱڀݚ‬͸, ؔઅͷॏཁ౓ͱؔ܎ੑΛߟྀͯ͠ಈ࡞ೝࣝΛߦ͏ Spatial Temporal Attention GCN ΛఏҊͨ͠. ·ͨ,ϚϧνϞʔμϧֶशʹ͓͚ΔϞʔμϧͷಛੑͷҧ͍ʹ΋ͱ͍ͮͯωοτϫʔΫߏங͢Δ Mechanics-stream ߏ଄ΛఏҊͨ͠. ධՁ࣮‫Ͱݧ‬͸, ఏҊख๏͕ैདྷख๏Λ௒͑Δೝࣝਫ਼౓Λୡ੒͠༗ޮੑΛ֬ೝͨ͠. ·ͨ, ಈ࡞ಛ༗ͷ Attention
graph ͕֫ಘ͢Δ͜ͱΛ֬ೝͨ͠. ࠓ‫ޙ‬ͷ՝୊ͱͯ͠, ಈ࡞ೝࣝҎ֎ͷλεΫ΁ͷԠ༻ͳͲ͕͋͛ΒΕΔ.

ࢀߟจ‫ݙ‬[1] S. Yan, et al., “Spatial Temporal Graph Convolutional Networks for Skeleton-Based Action Recognition”,
AAAI, 2018.
[2] S. Amir, et al., “NTU RGB+D: A Large Scale Dataset
for 3D Human Activity Analysis”, CVPR, 2016.
[3] L. Jun, et al., “NTU RGB+D 120: A Large-Scale Benchmark for 3D Human Activity Understanding”, TPAMI,
2019.

‫੷ۀڀݚ‬[1] K. Shiraki, et al., “Spatial Temporal Attention Graph
Convolutional Networks with Mechanics-Stream for
Skeleton-based Action Recognition”, ACCV, 2020.
(ଞ 3 ݅)Study of 3D Object Detection with Normal-map on Point Clouds
Supervisor: Takayoshi Yamashita
TP18051 Jishu Miao
1.Introduction
Object detection is one of the most crucial tasks inautonomous driving. In this task, both accuracy andspeed are important. There are various methods havebeen studied to improve accuracy while accelerating thedetection speed. Although RGB images captured bycameras are used for object detection, point clouds captured by LiDAR are also used for this task. Because
it is insensitive to visible light and can capture objects
day or night. However, point clouds are sparse and different from images in that the order is irregular, leading
to a slow processing speed as 2D convolution cannot
be performed directly. Surface normals extracted fromthe points have a better ability to represent the shapefeatures of the object than 3D coordinates, which we believe will improve the performance of object detection.
In this study, we propose a novel point clouds-based 3D
object detection method for achieving higher-accuracy.
The proposed method employs You Only Look Once
v4 (YOLOv4) as a feature extractor and gives Normalmap as additional input. Our Normal-map is a threechannels Bird-eye view (BEV) map, retaining detailed
surface normal vectors. It makes the input information
have more enhanced spatial shape information and canbe associated with other hand-crafted features easily.

2.Bird-eye View Representation
The BEV map represents the point clouds as a 2D
pseudo-image from the bird-eye view as shown in Figure
1. This approach converts the unordered point clouds
into a sequence ordered image. Conventional methods are to generate three maps by a mapping function
PΩi→j , representing normalized point cloud density (R),maximum height (G), and maximum reflection intensity(B), aszr (Sj ) = min(1.0,! log(N + 1)/64)" N = |PΩi→j | ,zg (Sj ) = max PΩi→j · [0, 0, 1]T ,zb (Sj ) = max (I (PΩi→j )) .

(1)

Since 2D convolution can be applied to the BEV map,the detection task can be accelerated by using a fast
object detection network such as YOLO[1]. However,different points may be arranged to a same pixel in the
BEV map, which is less expressive than original data.
Besides, the object shape will be lost due to the 3D datais compressed to 2D.

Figure 2 : Overview of Proposed Method.
normals. The mapping function fPS shown in Eq. (2)is used for creating the Normal-map, which allows us to
use the normal vector !xɼ!
y ɼ!z of the highest point inPΩi→j when mapping each point into 2D space.
#$PΩi→j = PΩi = [x, y, z]T |Sj = fPS (PΩi , g)
(2)

As shown in Eq. (3), the normal vectors of each point
are extracted as normal!x ɼnormaly! ɼnormal!z from thePΩi→h for each axis. The normal vectors are represented
by a 3-channel 2D pseudo image, asnormal!x (Sj ) = !x (PΩj→h ) ,normaly! (Sj ) = !
y (PΩj→h ) ,normal!z (Sj ) = !z (PΩj→h ) .

(3)

Since the search range of the normal estimation is wider
than the pixel representation range of the BEV map,
it can include a wider range of information. Moreover,since the normal-map calculated from the normal vectors is also a BEV map, it can be freely combined withother BEV maps to be used as the input data.

3.2 Input Details
Our network uses RGB-map and Normal-map as input. The RGB-map is similar to the BirdNet[2], andconsists of the height map, the density map, and theintensity map. The Normal-map is the normal vectors
in the x, y, and z axes. Thus, the input is a 6-channel
BEV map consisting of these maps concatenated in thechannel axis.

3.3 Object Detection Network
The network predicts the class and size of an object
with the Euler-Region Proposal Network (E-RPN) for3D object detection as shown in Figure 3. We employ
YOLOv4 as the basis network for 3D object detection.
E-RPN predicts the height, width, angle, objectness,and class probability of the bounding box coordinates.
In this study, the number of object classes is 6: Car,
Van, Truck, Person, Cyclist, and Tram. The loss function is based on Mean-Square Error.

Figure 1 : Bird-eye View Representation.

3.Proposed Method
We propose a method for 3D object detection using
BEV map with additional normal information. Figure
2 shows an overview of the proposed method.

3.1 Normal Feature Extraction
The normal vector is estimated from the pre-processed
point clouds by Principal Component Analysis (PCA)
with the search radius of 30cm and the maximum searchnumber of 50 points. To make all normals point in thesame direction, the normals opposing the LiDAR arereversed by an orienting system. Normal-map is generated for enabling the 2D convolution of each point’s

Figure 3 : Normal-YOLO Network Architecture.

4.Evaluation Experiments
In order to examine the effectiveness of the proposed
method, some comparison experiments are conductedusing the KITTI dataset. We also evaluate the accuracyof the object angle detection by adding a function to
calculate the yaw angle.

Table 1 : Evaluation Results for Bird-eye View Performance on the KITTI Benchmark.
MethodBirdNetComplexer-YOLO
Ours

FPS
9.1
16.7
5.5

Easy
84.17
77.24
72.84

Car
Mod.
59.83
68.96
71.52

Hard
57.35
64.95
67.50

Easy
28.20
21.42
26.71

Pedestrian
Mod.
Hard
23.06
21.65
18.26
17.06
21.19
20.17

Easy
58.64
32.00
42.50

Cyclist
Mod.
41.56
25.40
36.06

Hard
36.94
22.88
31.18

Average
45.93
38.68
43.30

Figure 5ɿ3D Object Detection Visualization in Camera View.

4.1 KITTI Benchmark Evaluation Results
Table 1 shows the evaluation results. Compared withthe conventional method, the proposed method achieves
the detection accuracy of 72.84% in Car under the Easy
mode, and the highest accuracy of 67.50% at the Hard
mode. It achieves almost the same accuracy as BirdNet[2],which is also a BEV-based method. Even for the sameclass objects with different detection modes, our method
shows a more robust performance by adding normal information. In addition, we achieve a higher Average
Precision (AP) than Complexer-YOLO[3], which uses
the same YOLO-based network. Although the input
is BEV map, the accuracy for non-planar objects (e.g.
pedestrians and cyclists) is better for each mode.

4.2 Evaluation of Angle Prediction
Since the normal is the object shape information, we
assume that object angle accuracy can be improved byadding the Normal-map. Table 2 shows the result of
calculating the average included angle θk from the estimated object angles and ground truth for 6 classes.
%
'−1
n
1&
scoreclass (θk ) =arccos θk
(4)
n
k=1

Normal-map improves the angle accuracy. In particular,
the accuracy of objects with large and flat shapes likecars is further improved.
Table 2 : Yaw Angle Prediction of 6 Classes.
Input Map
RGB
NormalNormal+RG
Normal+RGB

Car
10.18
8.76
9.27
10.34

Score of Angle Accuracy
Van
Truck
Person Cyclist
7.49
5.78
2.22
4.44
5.37
5.35
1.69
3.10
6.43
10.09
2.14
3.34
6.57
8.89
2.06
3.86

Tram
3.24
3.28
3.55
5.25

Figure 4 : Average Precision over distance.

4.4 Visualization Results
As shown in Figure 5, due to traffic lights, and traffic
signs are cylindrical and have a height similar to a human, the no-normal group could easily make false positive prediction. In contrast, proposed method reduce
the number of such mistakes with the addition of normal information. This indicates that the Normal-map isuseful in avoiding the false detection of objects similar
to human features.

5.Conclusion
We proposed a novel 3D object detection method with
Normal-map on point clouds. We have confirmed thatthe accuracy of BEV map-based object detection is further improved when we introduced normal information
to the BEV map. Since the Normal-map can keep highdetection accuracy without intensity information, it ispossible to use synthesized datasets by simulator withthe virtual environment. In the future, to improve theaccuracy of the method, we would like to explore deeplearning methods for object normal estimation.

References[1] A. Bochkovskiy, et al., “YOLOv4: Optimal Speed andAccuracy of Object Detection”, arXiv, 2020.
[2] J. Beltran, et al., “BirdNet: a 3D Object Detection
Framework from Lidar Information”, ITSC, 2018.

4.3 Evaluation by Distance
Since the density of the point clouds changes depending on the distance, we evaluate the detection accuracy
by distance. Figure 4 shows average precision over distance. From Figure 4, the detection accuracy of thegroup with added Normal-map does not decrease up to
30m, and the decrease is smaller than no-normal groupeven at 40m or more.

[3] M. Simon, et al., “Complexer-YOLO: Real-time 3D Object Detection and Tracking on Semantic Point Clouds”,
CVPR, 2019.

Research Achievements[1] J. Miao, et al., “3D Object Detection with Normal-map
on Point Clouds”, VISAPP, 2021.
(Other: 1 conference presentation)

Attention ‫ʹߏػ‬ΑΔࢹ֮తઆ໌ΛՄೳͱͨ͠ਂ૚‫ڧ‬Խֶशʹؔ͢Δ‫ڀݚ‬ࢦಋ‫ڭ‬तɿࢁԼོٛ
TP19001 ൘୩ӳయ
1. ͸͡Ίʹ

ΈࠐΈ૚ͱ Sigmoid ؔ਺Λద༻͢Δ͜ͱͰ֫ಘ͢Δɽਂ૚‫ڧ‬Խֶश͸ɼΤʔδΣϯτ͕‫ͱڥ؀‬ͷΠϯλϥΫ
γϣϯΛ௨ͯ͡ಘΔใुΛཔΓʹɼঢ়‫ʹگ‬Ԡͯ͡࠷దͳߦಈબ୒͢ΔͨΊͷํࡦΛֶश͢Δख๏Ͱ͋Δɽํࡦ͸ωοτϫʔΫͰද‫͞ݱ‬Ε͓ͯΓɼωοτϫʔΫ಺෦ͷԋࢉ͸ෳࡶͰ͋ΔɽͦͷͨΊɼΤʔδΣϯτͷߦಈબ୒ʹରͯ͠൑அࠜ‫ڌ‬Λղੳ͢Δ͜ͱ͸ඇৗʹࠔ೉Ͱ͋Δɽຊ‫Ͱڀݚ‬͸ɼਂ૚‫ڧ‬Խֶशͷ୅දతͳख๏Ͱ͋Δ Asynchronous Advantage Actor-Critic (A3C) [1] ʹ Attention‫ߏػ‬Λಋೖͨ͠ Mask-Attention A3C (Mask A3C) ΛఏҊ͢ΔɽMask A3C Ͱ͸ɼํࡦͱঢ়ଶՁ஋ʹର͢ΔΤʔδΣϯτͷ஫ࢹྖҬΛද‫ ͨ͠ݱ‬Mask-attention Λ֫ಘͰ͖Δɽ
ਪ࿦࣌ʹ Mask-attention ΛՄࢹԽ͢Δ͜ͱͰɼΤʔδΣϯτͷߦಈબ୒ʹର͢Δࢹ֮తઆ໌ͷ࣮‫ݱ‬Λ໨తͱ͢Δɽ
·ͨɼAttention ‫ߏػ‬ͷಋೖʹΑΓɼMask-attention Λߟֶྀͯ͠श͢Δ͜ͱͰɼΤʔδΣϯτͷੑೳ޲্ΛਤΔɽ

3.2 Attention ‫ߏػ‬

2.Asynchronous Advantage Actor-Critic

Mask A3C ͷ༗ޮੑΛ֬ೝ͢ΔͨΊɼOpenAI Gym [3]ͷήʔϜλεΫΛ༻͍ͯධՁ࣮‫ݧ‬Λߦ͏ɽ

A3C [1] ͸ɼAsynchronous ͱ Advantage Λಋೖͨ͠
Actor-Critic ๏ϕʔεͷਂ૚‫ڧ‬Խֶशख๏Ͱ͋ΔɽAsynchronous ͸ෳ਺‫͚͓ʹڥ؀‬Δඇಉ‫Ͱظ‬ͷύϥϝʔλߋ৽ɼ
Advantage ͸਺εςοϓઌͷใुΛߟֶྀͨ͠शͰ͋Δɽ
A3C ͷωοτϫʔΫߏ଄͸ɼ৞ΈࠐΈ૚ʹΑΓಛ௃ϚοϓΛநग़͢Δ Feature extractorɼํࡦΛग़ྗ͢Δ Policy
branchɼঢ়ଶՁ஋Λग़ྗ͢Δ Value branch ͔Βߏ੒͞Ε
Δɽํࡦ͸͋Δঢ়ଶʹ͓͍ͯબ୒͢Δߦಈͷ֬཰෼෍Ͱ͋
Δɽঢ়ଶՁ஋͸͋Δঢ়ଶʹ͓͚Δใुͷ‫ظ‬଴஋Ͱ͋Γɼ͋Δঢ়ଶʹ͍Δ͜ͱͷՁ஋Λද͢ɽ

3.Mask-Attention A3C
ΤʔδΣϯτͷߦಈબ୒ʹର͢Δ൑அࠜ‫ڌ‬Λղੳ͢Δ
ͨΊɼA3C ʹ Attention ‫ߏػ‬Λಋೖͨ͠ Mask-Attention
A3C (Mask A3C) ΛఏҊ͢ΔɽMask A3C Ͱ͸ɼPolicy
branch ͱ Value branch ʹର͠ Attention ‫ߏػ‬Λಋೖ͢
Δ͜ͱͰɼ֤ϒϥϯνͷग़ྗʹରͯ͠஫ࢹͨ͠ྖҬΛද͢
Mask-attention Λ֫ಘ͢Δɽਪ࿦࣌ʹ͓͚Δ֤ϒϥϯνͷ Mask-attention ΛՄࢹԽ͢Δ͜ͱͰɼํࡦͱঢ়ଶՁ஋
ͷҟͳΔ 2 ͭͷࢹ఺͔ΒɼΤʔδΣϯτͷߦಈબ୒ʹର͢
Δࢹ֮తઆ໌Λ࣮‫͢ݱ‬Δɽ

3.1 ωοτϫʔΫߏ଄

Conv.

Sigmoidਤ 1 ʹఏҊ͢Δ Mask A3C ͷωοτϫʔΫߏ଄Λࣔ͢ɽ
Mask A3C ͸ɼFeature extractorɼPolicy branchɼValue
branchɼAttention ‫͔ߏػ‬Βߏ੒͞ΕΔɽैདྷͷ A3C Ͱ͸ɼ࣌‫ྻܥ‬৘ใΛߟྀ͢ΔͨΊɼFeature extractor ʹ LSTM
Λ༻͍Δɽ͔͠͠ɼMask A3C ʹ LSTM Λಋೖ͢Δͱɼೖྗը૾ʹର͢Δۭؒ৘ใ͕ܽམ͢ΔͨΊɼMask-attentionΛ֫ಘͰ͖ͳ͍ɽͦ͜Ͱɼۭ࣌ؒ৘ใΛߟྀͰ͖Δ Convolutional LSTM (ConvLSTM) [2] Λಋೖ͢Δɽϒϥϯ
ν i ͷ Mask-attention Mi ͸ɼFeature extractor ͔Βग़ྗ͞ΕΔಛ௃Ϛοϓ Fi ʹର͠ɼ1 × 1× # of chanels ͷ৞#$FCState value

FC

Conv.

&$'

&$

PolicyPolicy branch
Conv.

ReLU

Conv.

Attentionmechanism

Sigmoid

Input

Value branch

ConvLSTM

Conv.

Maxpooling + ReLU

Conv.

Attentionmechanism
Maxpooling + ReLU

Conv.

Maxpooling + ReLU

!"

ReLU

Mask-attention

&%'

&%

Feature extractor#%
Mask-attention

ਤ 1: Mask A3C ͷωοτϫʔΫߏ଄

Mask A3C Ͱ͸ɼPolicy branch ͱ Value branch ʹ
Attention ‫ߏػ‬Λಋೖ͢Δɽ͜ΕʹΑΓɼ֫ಘͨ͠ Maskattention Mi Λߟྀ͠ɼํࡦ‫ͼٴ‬ঢ়ଶՁ஋Λग़ྗ͢ΔɽAttention ‫ߏػ‬͸ɼϒϥϯν i ಺ʹ͓͚Δதؒ૚ͷಛ௃ϚοϓFi ʹର͠ɼMask-attention Mi Λ༻͍ͯϚεΫॲཧΛߦ͏ɽಛ௃Ϛοϓʹର͢Δ Mask-attention Λ༻͍ͨϚεΫॲཧΛࣜ (1) ʹࣔ͢ɽ͜͜Ͱɼst ͸ೖྗͰ͋Δ‫ݱ‬ঢ়ଶɼFi (st )͸ϒϥϯν i ಺ʹ͓͚Δதؒ૚ͷಛ௃ϚοϓɼMi (st ) ͸ϒϥϯν i ʹ͓͚Δ Mask-attentionɼFi! (st ) ͸ϚεΫॲཧ‫ޙ‬ͷಛ௃ϚοϓͰ͋Δɽ
Fi! (st ) = Fi (st )ɾMi (st )(1)

4. ධՁ࣮‫ݧ‬

4.1 ࣮‫֓ݧ‬ཁ࢖༻͢ΔήʔϜ͸ɼBreakout ͱ Ms.Pac-ManɼSpace
Invaders ͷ 3 छྨͰ͋Δɽൺֱख๏͸ɼA3C ͱ Mask A3Cɼ֤ϒϥϯνͷΈʹରͯ͠ Attention ‫ߏػ‬Λಋೖͨ͠ Mask
A3C (Policy Mask A3CɼValue Mask A3C) ͷ‫ ܭ‬4 ͭͰ
͋Δɽೖྗ͸ήʔϜը໘ͷάϨʔεέʔϧը૾ͱ͠ɼग़ྗ
Ͱ͋Δߦಈ͸֤ήʔϜʹ͓͚Δૢ࡞ͱ͢Δɽֶश৚݅͸
worker ਺Λ 35ɼֶश܎਺Λ 0.0001ɼׂҾ཰Λ 0.99 ͱ͢Δɽֶशऴྃ৚݅͸ global step ਺͕ 1.0 × 108 ʹ౸ୡͨ͠
৔߹ͱ͢Δɽ·ͨɼΤϐιʔυͷऴྃ৚݅͸֤ήʔϜʹ͓
͚Δ 1 ϓϨΠऴྃɼ‫ ͼٴ‬step ਺͕ 1.0 × 104 ʹ౸ୡͨ͠৔
߹ͱ͢Δɽ

4.2 είΞൺֱ֤ήʔϜλεΫʹ͓͚Δ 100 Τϐιʔυؒͷ࠷େ/ฏ‫ۉ‬είΞΛද 1 ʹࣔ͢ɽද 1 ͔ΒɼBreakout ʹ͓͚Δ࠷େείΞ͸શख๏ʹ͓͍ͯ 864 Ͱ͋Δɽ͜ͷείΞ͸ɼBreakoutͰ֫ಘͰ͖Δ࠷ߴείΞͰ͋Δɽ·ͨɼBreakout ʹ͓͚Δฏ‫ۉ‬είΞ͸ɼMask A3C ͕ A3C ͱൺֱ͠΄΅ಉ౳Ͱ͋ΔɽMs.Pac-Man Ͱ͸ɼ࠷େ/ฏ‫ۉ‬είΞ‫ ʹڞ‬Mask A3C͕࠷΋ߴ͍είΞͰ͋ΔɽSpace Invaders Ͱ͸ɼ࠷େείΞ͸ Policy Mask A3Cɼฏ‫ۉ‬είΞ͸ Mask A3C ͕࠷΋ߴ͍είΞͰ͋ΔɽBreakout ͸ύυϧͰϘʔϧΛଧͪฦ͢ͷΈͰ͋Γɼ֎తཁҼͷͳ͍୯७ͳλεΫͰ͋ΔɽͦͷͨΊɼA3C ͱ Mask A3C ͕ಉ౳ͷείΞͰ͋ͬͨͱߟ͑ΒΕΔɽҰํɼMs.Pac-Man ͱ Space Invaders ͸ɼఢͳͲͷ֎తཁҼΛߟྀͯ͠ߦಈΛબ୒͢Δඞཁ͕͋ΔɽPolicy
branch ʹ Attention ‫ߏػ‬Λಋೖͨ͠ Policy Mask A3C ͱ
Mask A3C Ͱ͸ɼΫοΩʔ΍ΠϯϕʔμʔͳͲɼํࡦʹؔ࿈ͨ͠ྖҬΛ‫ڧ‬ௐ͢ΔɽͦͷͨΊɼA3C ͱൺֱ͠είΞ͕޲্ͨ͠ͱߟ͑ΒΕΔɽ

4.3 Mask-attention Λ༻͍ͨࢹ֮తઆ໌
֤ήʔϜʹ͓͚Δ Mask-attention ͷՄࢹԽྫΛਤ 2 ʹࣔ͢ɽਤ 2(a) ͷ Policy ͔ΒɼFrame 1 Ͱ͸Ϙʔϧͷਐߦํ޲Λ஫ࢹ͍ͯ͠ΔɽFrame 2 Ͱ͸ύυϧΛ‫ؚ‬ΊͨϘʔϧपғΛ஫ࢹ͠ɼϘʔϧΛଧͪฦͨ͠‫͋Ͱޙ‬Δ Frame 3 Ͱ͸஫ࢹ͍ͯ͠ΔྖҬ͕ͳ͍ɽ͔͜͜ΒɼBreakout Ͱ͸ϘʔϧΛଧͪฦ͢௚લʹ͓͍ͯɼϘʔϧͷਐߦํ޲Λ༧ଌ͠ύυϧΛ੍‫͍ͯ͠ޚ‬Δͱߟ͑ΒΕΔɽਤ 2(b) ͷ Policy ͔ΒɼFrame 1 Ͱ͸ύοΫϚϯपғΛ஫ࢹ͍ͯ͠ΔɽFrame
2 Ͱ͸ը໘಺ʹଘࡏ͢ΔΫοΩʔΛ஫ࢹ͠ɼFrame 3 Ͱ͸ Frame 2 ʹ͓͚Δ஫ࢹྖҬʹύοΫϚϯ͕Ҡಈ͍ͯ͠Δɽ͔͜͜ΒɼMs.Pac-Man Ͱ͸ύοΫϚϯͷपғΛ‫ܯ‬ռ͠ͳ͕ΒɼΫοΩʔ΁޲͔͏Α͏ʹ੍‫͍ͯ͠ޚ‬Δͱߟ͑ΒΕΔɽਤ 2(c) ͷ Policy ͔ΒɼFrame 1 ͰΠϯϕʔμʔΛ஫ࢹ͠ɼߦಈ͸߈ܸΛબ୒͍ͯ͠Δɽͦͯ͠ɼFrame 3

Point 1

Frame 1

Frame 2

Point 2

Point 1

Frame 3

PolicyPoint 3

Point 2

Point 3

State value

State value graph(a) Breakout: ന໼ҹ͸Ϙʔϧͷਐߦํ޲Λࣔ͢ɽ
Point 1

Frame 1

Frame 2

Point 1

Frame 3

Point 3

Point 3

Point 2

PolicyPoint 2

State value graphState value(b) Ms.Pac-Man: ന໼ҹ͸ύοΫϚϯͷਐߦํ޲Λࣔ͢ɽ
Point1

Frame 1

Frame 2

Point 1

Frame 3

Point 2

PolicyPoint2

ɹ

Point3

Point 3

State value graphState value(c) Space Invaders: ന໼ҹ͸ϏʔϜͷਐߦํ޲Λࣔ͢ɽਤ 2: Mask-attention ͷՄࢹԽྫ: Policy ʹ͓͚ΔԼ෦ͷίϯτϩʔϥ͸ɼ‫ݱ‬ϑϨʔϜͰϞσϧ͕ਪ࿦ͨ͠ߦಈͰ͋Δɽද 1: ֤ήʔϜ͓͚Δ 100 Τϐιʔυؒͷ࠷େ/ฏ‫ۉ‬είΞ:֤ख๏Ͱ 5 ࢼߦֶͣͭश͠ɼฏ‫ۉ‬είΞ͕࠷΋ߴ͍ϞσϧͷείΞΛࣔ͢ɽmax/mean ͸࠷େ/ฏ‫ۉ‬είΞͰ͋Δɽ
Ms.Pac-Man

Space Invaders

Att. mechanism

max

mean

max

mean

max

mean

Policy

864

662.0

5380

4573.3

19505

18531.8

864

595.8

6330

4833.819860

19102.8

!

864

606.9

4830

4044.5

19675

18537.8

!

864

640.0

6610

5314.1

19810

19212.5

Att. mechanism
Policy

Value

A3C
!
ఏҊख๏

!

ද 2: Mask-attention ͷ൓సʹΑΔείΞൺֱɿnormal ͸ Mask-attention Λ൓స͠ͳ͍৔߹ɼin- ɹverse ͸൓సͨ͠৔߹ɼrandom ͸ϥϯμϜʹߦಈબ୒ͨ͠৔߹Ͱ͋Δɽ

Breakoutʹ͓͍ͯ Frame 1 Ͱ஫ࢹͨ͠ΠϯϕʔμʔΛܸഁ͍ͯ͠
Δɽ͔͜͜ΒɼSpace Invaders Ͱ͸ܸഁ͢ΔΠϯϕʔμʔΛೝࣝ͠ɼΤʔδΣϯτΛ੍‫͍ͯ͠ޚ‬Δͱߟ͑ΒΕΔɽ·ͨɼਤ 2(a)(b)(c) ͷ State value ͔ΒɼBreakout Ͱ͸ϒϩοΫɼMs.Pac-Man Ͱ͸ΫοΩʔɼSpace Invaders Ͱ͸ΠϯϕʔμʔΛ஫ࢹ͍ͯ͠Δɽͦͯ͠ɼ஫ࢹͨ͠෺ମͷ‫ݮ‬গʹ߹Θͤ஫ࢹྖҬ͕ॖখ͍ͯ͠Δɽ͜ΕΒͷ݁Ռ͔Βɼ
Policy ͷ Mask-attention ͸‫ݱ‬ঢ়ଶʹର͠ߦಈʹ௚݁͢Δ
෺ମɼState value ͷ Mask-attention ͸είΞʹ‫د‬༩͢Δ
෺ମΛද͍ͯ͠Δͱߟ͑ΒΕΔɽ

4.4 Mask-attention ͷ༗ޮੑ
Mask A3C ʹΑΔߦಈͷࢹ֮తઆ໌Λߦ͏ʹ͋ͨΓɼ
Mask-attention ͕Ϟσϧͷग़ྗͰ͋Δํࡦʹରͯ͠༗ӹͳྖҬΛද͍ͯ͠Δ͔‫͢ূݕ‬Δɽ‫ํূݕ‬๏ͱͯ͠ɼMask
A3C ʹ͓͚Δ Policy branch ͷ Mask-attention Λ൓సͨ͠ϚοϓΛ࡞੒͠ɼͦͷϚοϓΛ Attention ‫͍ͨ༻ʹߏػ‬৔߹ͷείΞΛࢉग़͢ΔɽMask-attention Λ൓స͢Δ৔߹
ͱ൓స͠ͳ͍৔߹ʹ͓͚ΔείΞΛൺֱ͢Δ͜ͱͰɼMaskattention ͕ߦಈͷࢹ֮తઆ໌ʹ༗ޮͰ͋Δ͜ͱΛ֬ೝ͢
Δɽද 2 ʹɼMask-attention ͷ൓సʹΑΔείΞൺֱΛࣔ
͢ɽද 2 ͔ΒɼશήʔϜʹ͓͍ͯ inverse ͕ normal ͱൺֱͯ͠ஶ͘͠είΞ͕௿Լ͠ɼrandom ͱಉ౳ͷείΞͰ͋Δɽ͕ͨͬͯ͠ɼPolicy branch ʹ͓͚Δ Mask-attention

normal!
!

Breakout

Value!
random

Ms.Pac-Man

Space Invaders

max

mean

max

mean

max

mean

864

595.8

6630

4833.819860

19102.8

inverse

4

2.2

290

268.9

805

306.9

normal

864

640.0

6610

5314.1

19810

19212.5

inverse

5

1.8

410

194.4

915

420.2

5

1.2

1080

247.8

460

142.1

͸ɼߴείΞΛ֫ಘ͢Δߦಈʹରͯ͠༗ӹͳྖҬΛ֫ಘͨ͠ͱ‫͑ݴ‬Δɽ

5. ͓ΘΓʹ
ຊ‫Ͱڀݚ‬͸ɼਂ૚‫ڧ‬Խֶशͷ୅දతͳख๏Ͱ͋Δ A3C ʹ
Attention ‫ߏػ‬Λಋೖͨ͠ Mask A3C ΛఏҊͨ͠ɽMask
A3C Ͱ͸ɼਪ࿦࣌ʹ Mask-attention ΛՄࢹԽ͢Δ͜ͱͰɼΤʔδΣϯτͷߦಈબ୒ʹର͢Δ൑அࠜ‫ڌ‬ͷࢹ֮తઆ໌Λ࣮‫ͨ͠ݱ‬ɽࠓ‫ޙ‬͸ɼϩϘοτ੍‫Ͳͳޚ‬ͷߋʹෳࡶͳλεΫ΁ͷద༻ͱՄࢹԽʹऔΓ૊Ή༧ఆͰ͋Δɽࢀߟจ‫ݙ‬[1] V. Minh, et al., “Asynchronous Methods for Deep Reinforcement Learning”, ICML, 2016.
[2] X. Shi, et al., “Convolutional LSTM Network: A Machine Learning Approach for Precipitation Nowcasting”,
NeurIPS, 2015.
[3] G. Brockman, et al., “OpenAI Gym”, arXiv preprint
arXiv:1606.01540, 2016.

‫੷ۀڀݚ‬[1] ൘୩ӳయ ౳, “A3C ʹ͓͚Δ Attention ‫ߏػ‬Λ༻͍ͨࢹ֮త
આ໌”, ਓ޻஌ೳֶձશࠃେձ, 2020. (ֶੜ঑ྭ৆ड৆)(ଞ 2 ݅)ηϚϯςΟοΫηάϝϯςʔγϣϯʹΑΔ௒ߴղ૾౓ը૾͔ΒͷมԽ఺‫ݕ‬ग़ʹؔ͢Δ‫ڀݚ‬ࢦಋ‫ڭ‬तɿࢁԼོٛ

TP19015 ౵Ҫॣ‫ޗ‬

2. มԽ఺‫ݕ‬ग़ͷैདྷ๏
CNN Λ༻͍ͨมԽ఺‫ݕ‬ग़๏͸ɼಛ௃ྔϕʔεͱֶशϕʔεͷ 2 ͭͷΞϓϩʔνʹ෼ྨͰ͖Δɽ
ಛ௃ྔϕʔεͷมԽ఺‫ݕ‬ग़๏ɿCNN ΑΓநग़ͨ͠ 2 ຕͷಛ௃Ϛοϓͷ‫͔ࢉܭ཭ڑ‬ΒมԽྖҬΛਪఆ͢ΔɽͦͷͨΊɼมԽ఺‫ݕ‬ग़ͷͨΊͷֶशσʔληοτΛඞཁͱ͠ͳ͍ɽಛ௃
ྔϕʔεͷ୅දతͳख๏Ͱ͋Δ CNNF-F[1] ͸ɼImageNetͰࣄલֶशͨ͠ VGG19 Λ༻͍ͯɼը૾ϖΞ͔Βಛ௃Ϛοϓநग़͠มԽྖҬΛ‫ݕ‬ग़͢Δɽ‫ݕ‬ग़ʹ͸ϢʔΫϦου‫཭ڑ‬Λ࢖༻͠ɼมԽྖҬΛܾఆ͢ΔͨΊʹᮢ஋ॲཧΛࢪ͢ɽมԽྖҬͷΈΛ‫ݕ‬ग़͢Δख๏Ͱ͋ΔͨΊɼมԽྖҬʹଐੑΛ෇༩͢Δ͜ͱ͕Ͱ͖ͳ͍ɽ
ֶशϕʔεͷมԽ఺‫ݕ‬ग़๏ɿมԽ఺‫ݕ‬ग़ͷͨΊͷσʔληοτΛ༻͍ͯը૾ؒͷྨࣅੑ (ඇྨࣅੑ) Λֶश͢ΔͨΊɼલड़ͷΞϓϩʔνͱൺ΂ͯࢹ఺ͷͣΕʹ‫ݕͳ݈ؤ‬ग़͕ՄೳͰ
͋Δɽֶशϕʔεͷ୅දతͳख๏Ͱ͋Δ FC-EF-Res[2] ͸ɼมԽ఺‫ݕ‬ग़λεΫͱηϚϯςΟοΫηάϝϯςʔγϣϯλεΫΛͦΕͧΕผʑͷωοτϫʔΫͰॲཧ͠ɼͦͷ݁ՌΛ౷߹͢Δख๏Ͱ͋Δɽ̎ͭͷωοτϫʔΫ͸ U-Net[3] Λϕʔεʹߏ੒͞Εɼ࢒ࠩϒϩοΫΛಋೖ͍ͯ͠ΔɽηάϝϯςʔγϣϯωοτϫʔΫ͔Βநग़ͨ͠ಛ௃Ϛοϓͷࠩ෼ΛɼมԽ఺‫ݕ‬ग़ωοτϫʔΫʹεΩοϓ઀ଓ͢Δ͜ͱͰɼը૾ؒͷ໌ࣔతͳൺֱΛߦ͍ͳ͕Βֶश͢Δɽ͔͠͠ɼλεΫ͝ͱʹಠཱͨ͠ωοτϫʔΫΛ༻͍ΔͨΊɼֶशதʹλεΫؒͰ৘ใΛ૬‫ޓ‬ར༻͢Δ͜ͱ͕Ͱ͖ͳ͍ͱ͍͏໰୊͕͋Δɽ

3. ఏҊख๏
ຊ‫Ͱڀݚ‬͸ɼมԽྖҬͱΦϒδΣΫτΫϥεΛಉ࣌ʹਪఆ͢Δ͜ͱ͕Ͱ͖ΔมԽ఺‫ݕ‬ग़๏ΛఏҊ͢ΔɽఏҊख๏ͷωοτϫʔΫߏ଄Λਤ 1 ʹࣔ͢. ωοτϫʔΫ͸ɼEncoderDecoder ߏ଄Λ࣋ͭ̎ͭͷϞσϧ͔Βߏ੒͞Ε͓ͯΓɼมԽ఺‫ݕ‬ग़Ϟσϧʹ͸ηάϝϯςʔγϣϯϞσϧͷ࠷ऴ૚͔Βநग़ͨ͠ಛ௃ϚοϓΛೖྗ͢Δɽ

3.1 ηάϝϯςʔγϣϯϞσϧ
ηάϝϯςʔγϣϯϞσϧ͸ɼU-Net Λϕʔεͱ͢ΔωοτϫʔΫߏ଄Ͱ͋Δɽը૾ϖΞͷͦΕͧΕΛ‫ݸ‬ผʹωοτϫʔΫ΁ೖྗ͢Δɽͳ͓ɼ֤ωοτϫʔΫͷॏΈ͸‫ڞ‬༗͍ͯ͠ΔɽηάϝϯςʔγϣϯϞσϧͷଛࣦؔ਺͸ɼैདྷͷ
U-Net ͱಉ༷ɼΫϩεΤϯτϩϐʔଛࣦΛ࢖༻͢Δɽ

3.2 มԽ఺‫ݕ‬ग़ϞσϧมԽ఺‫ݕ‬ग़Ϟσϧ͸ɼ̎ͭͷ Encoder ͱ 1 ͭͷ DecoderͰߏ੒ͨ͠ωοτϫʔΫߏ଄ͱͳ͓ͬͯΓɼͦΕͧΕ 3 ૚ͷ৞ΈࠐΈ૚Λ࣋ͭɽ2 ͭͷ Encoder ʹΑΓɼը૾ؒͰҟ
ͳΔಛ௃Λ֫ಘ͢Δ͜ͱ͕ՄೳͰ͋Δɽ·ͨɼ֤ Encoderʹ͸ηάϝϯςʔγϣϯϞσϧΑΓಘΒΕͨಛ௃ϚοϓΛͦΕͧΕೖྗ͠ɼDecoder ʹ͸֤ Encoder ΑΓಘΒΕͨಛ௃ϚοϓΛνϟϯωϧํ޲ʹ࿈݁͠ೖྗ͢Δɽ

image (past)

softmax

Segmentation (past)weight share

Change detection
image (present)

Change detection model !$%Segmentation model !!"#

softmaxมԽ఺‫ݕ‬ग़͸ɼҟͳΔ࣌ࠁʹࡱӨ͞ΕͨಉҰγʔϯը૾͔ΒมԽͨ͠ྖҬΛநग़͢Δ໰୊Ͱ͋ΔɽCNN Λ༻͍ͨมԽ఺‫ݕ‬ग़๏ [1][2] ͸ɼҐஔͣΕ΍‫ڥ؀‬มԽʹ‫͋Ͱ݈ؤ‬Δ
ͨΊɼϚοϓͷࣗಈߋ৽΍౔஍ར༻ௐࠪ΁ͷར༻͕‫ظ‬଴͞Ε͍ͯΔɽ౔஍ར༻ௐࠪʹԠ༻͢Δ৔߹ɼมԽͨ͠ྖҬͷ‫ݕ‬ग़͚ͩͰͳ͘ɼ೶஍΍৿ྛͳͲա‫ࡏݱͱڈ‬ͷ౔஍ͷଐੑ৘ใ͕ඞཁͰ͋Δɽ͜ΕʹΑΓɼมԽͨ͠৔ॴͷ಺༁Λ೺Ѳ͢Δ͜ͱ͕Ͱ͖Δɽ
ͦ͜Ͱຊ‫Ͱڀݚ‬͸ɼมԽྖҬͱΦϒδΣΫτΫϥεΛಉ࣌ʹਪఆ͠ɼมԽྖҬʹ౔஍ͷଐੑ৘ใΛ෇༩͢Δ͜ͱ͕
Ͱ͖ΔมԽ఺‫ݕ‬ग़๏ΛఏҊ͢ΔɽఏҊख๏͸ɼηϚϯςΟο
ΫηάϝϯςʔγϣϯͱมԽ఺‫ݕ‬ग़Λ 1 ͭͷωοτϫʔΫͰ࣮‫͢ݱ‬Δɽ·ͨɼηάϝϯςʔγϣϯϞσϧΑΓநग़ͨ͠ಛ௃ϚοϓΛɼมԽ఺‫ݕ‬ग़Ϟσϧͷೖྗͱͯ͠࢖༻͢Δ
͜ͱͰɼ྆λεΫͷਫ਼౓޲্ΛਤΔɽ

softmax

1. ͸͡Ίʹ

Segmentation (present)ਤ 1 : มԽ఺‫ݕ‬ग़ωοτϫʔΫͷ֓ཁ

3.3 ஈ֊తͳֶशͱଛࣦؔ਺
ఏҊख๏͸ɼηάϝϯςʔγϣϯϞσϧͱมԽ఺‫ݕ‬ग़ϞσϧʹΑΓߏ੒͞Εɼஈ֊తͳֶशΛߦ͏ɽ
Step1ɿ·ͣɼϕʔεͱͳΔηάϝϯςʔγϣϯϞσϧ θsegΛֶश͢ΔɽϐΫηϧ૯਺Λ N ɼΫϥε਺Λ S, ਖ਼ղϥϕϧΛ tɼιϑτϚοΫεؔ਺ͷग़ྗΛ y ͱ͢ΔͱɼηϚϯςΟοΫηάϝϯςʔγϣϯλεΫͷଛࣦؔ਺ LSeg ͸ࣜ(1) Ͱද͢͜ͱ͕Ͱ͖Δɽ
N

LSeg = −S

1 !!
wc ti,c log yi,c
N i c

(1)

͜͜Ͱɼwc ͸֤λεΫͷॏΈͰ͋ΔɽӴ੕ը૾ͳͲͷ௒ߴղ૾౓ը૾Ͱߏ੒͞ΕΔσʔληοτΛ࢖༻͢Δ৔߹ɼΫϥεෆ‫͕ߧۉ‬ੜ͡Δɽϥϕϧ͕ෆ‫ߧۉ‬ͷ··ֶश͢Δͱɼग़‫཰֬ݱ‬ͷߴ͍Ϋϥεʹରͯ͠ૣ‫ʹظ‬ऩଋͯ͠͠·͍ɼग़‫཰֬ݱ‬ͷ௿͍Ϋϥε͕΄ͱΜͲ൓ө͞Εͳ͍໰୊͕͋ΔɽͦͷͨΊɼࣜ (2) ʹࣔ͢Α͏ʹɼग़‫ ཰֬ݱ‬pc ͷ‫ʹ਺ٯ‬ର਺ΛॏΈͱ͢Δɽ
1
wc = log(2)pc͜ΕʹΑΓɼग़‫཰֬ݱ‬ͷߴ͍ΫϥεʹΦʔόʔϑΟοςΟ
ϯά͢Δ͜ͱΛ๷͙ɽ
Step2ɿ࣍ʹɼηάϝϯςʔγϣϯϞσϧ θseg ͷॏΈΛ‫ݻ‬ఆ͠ɼมԽ఺‫ݕ‬ग़Ϟσϧ θcd Λֶश͢Δɽࣜ (2) Ͱఆٛͨ͠ॏΈΛ࢖༻͠ɼΫϥε਺Λ D ͱ͢ΔͱɼมԽ఺‫ݕ‬ग़λεΫͷଛࣦؔ਺ LDif f ͸ࣜ (3) Ͱද͢͜ͱ͕Ͱ͖Δɽ
N

LDif f = −D

1 !!
wc ti,c log yi,c
N i c

(3)

Step3ɿͦͯ͠ɼ2 ͭͷϞσϧͷ૬‫܎ؔޓ‬ΛอͭͨΊɼθsegͱ θcd Λಉ࣌ʹֶश͢Δɽ૬‫܎ؔޓ‬ΛอͭͨΊͷଛࣦؔ਺LM ut Λࣜ (4) ʹࣔ͢ɽStep2 ͱಉ༷ɼLM ut ͷࢉग़ʹ͸θcd ΑΓநग़ͨ͠ಛ௃ϚοϓΛ࢖༻͢ΔɽN

LM ut = −D

1 !!
ti,c log yi,c
N i c

(4)

Step4ɿ֤ iteration Ͱ Step1 ͔Β Step3 Λ‫܁‬Γฦ͢ɽҎ্ʹࣔ͢ஈ֊తͳֶशʹΑΓɼλεΫؒͰ৘ใΛ૬‫ޓ‬ར༻͠ɼ྆λεΫͷਫ਼౓޲্͕‫ظ‬଴Ͱ͖Δɽ

3.4 ֶशσʔλͷೖྗํ๏ఏҊख๏͸ɼ10,000×10,000 ϐΫηϧͷ௒ߴղ૾౓ը૾
Λର৅ͱ͠ɼֶश࣌͸ 512×512 ϐΫηϧʹϥϯμϜΫϩο
ϓ͢Δɽ͔͠͠ɼΫϥεෆ‫͕ߧۉ‬ੜ͡Δ৔߹ɼϥϯμϜΫ
ϩοϓͰ͸ग़‫཰֬ݱ‬ͷ௿͍Ϋϥε͕બ୒͞Εͣɼֶश͕ෆ҆ఆͱͳΔɽͦ͜Ͱɼग़‫཰֬ݱ‬ͷ௿͍ΫϥεΛଟ͘‫ؚ‬Ήύονը૾ΛผͰ༻ҙ͠ɼ௨ৗͷύονը૾ͱͱ΋ʹֶशΛߦ͏ɽ͜ΕʹΑΓɼΫϥεෆ‫ʹߧۉ‬ΑΔӨ‫ڹ‬Λ཈͑Δ͜ͱ͕Ͱ͖ΔͨΊɼֶशͷ҆ఆੑʹͭͳ͕Δɽ

4. ධՁ࣮‫ݧ‬ఏҊख๏ͷ༗ޮੑΛࣔͨ͢ΊʹɼมԽ఺‫ݕ‬ग़‫ͼٴ‬ηϚϯςΟοΫηάϝϯςʔγϣϯͷධՁ࣮‫ݧ‬Λߦ͏ɽද 2 : มԽ఺‫ݕ‬ग़ͱηϚϯςΟοΫηάϝϯςʔγϣϯͷൺֱ݁Ռ

Semantic segmentation
Change detection
Global Accuracy Class Accuracy mean IoU Kappa Global Accuracy mean IoU Kappa
Seg
90.96
62.78
45.83
78.56
92.36
48.28
6.86
U-Net
Diff
99.25
54.56
17.72
CNNF-F
88.66
3.28
FC-EF-Res
89.01
71.92
98.30
25.49
Proposed
91.46
63.21
53.48
79.24
99.17
59.32
32.20

image (2006)

image (2012)

Ground Truth

Proposedimage

ਤ 2 : ఏҊख๏ʹΑΔมԽ఺‫ݕ‬ग़ྫ

4.1 σʔληοτຊ࣮‫Ͱݧ‬͸ɼҟͳΔ࣌‫ࡱʹظ‬Ө͞ΕͨӴ੕ը૾ϖΞͰߏ੒͞ΕΔ HRSCD σʔληοτ [2] Λ࢖༻͢ΔɽHRSCDσʔληοτ͸ɼେ‫ن‬໛ͳ௒ߴղ૾౓ͷมԽ఺‫ݕ‬ग़༻σʔληοτͰ͋ΔɽRGB ը૾ϖΞͱɼ֤ϐΫηϧͰͷมԽ৘ใΛ෇༩ͨ͠มԽ‫ݕ‬ग़ϥϕϧ (มԽྖҬͳ͠ɼมԽྖҬ͋Γ)ɼ౔஍ར༻৘ใΛ෇༩ͨ͠ηάϝϯςʔγϣϯϥϕϧ
(৘ใͳ͠ɼਓ޻෺ɼ೶‫ۀ‬஍ଳɼ৿ྛɼ࣪஍ɼਫҬ) ͕‫·ؚ‬ΕΔɽσʔλ૯਺͸ 582 ຕ͋Γɼֶश༻ʹ 292 ຕ (146 ૊ͷը૾ϖΞ)ɼධՁ༻ʹ 290 ຕ (145 ૊ͷը૾ϖΞ) Λ࢖༻͢Δɽද 1 ʹ HRSCD σʔληοτͷ֤Ϋϥεʹ͓͚Δग़‫ͱ཰֬ݱ‬ॏΈΛࣔ͢ɽද 1 ΑΓɼग़‫͕཰֬ݱ‬খ͍͞΄ͲɼΫϥεॏΈ͕େ͖͘ͳΔ͜ͱ͕Θ͔Δɽද 1 : ֤Ϋϥεʹ͓͚Δग़‫ͱ཰֬ݱ‬ॏΈSemantic Segmentation

Ground Truth

U-Net (Seg)

Proposedਤ 3 : ֤ख๏ͷηάϝϯςʔγϣϯྫ

Change Detection

৘ใͳ͠

ਓ޻෺

೶‫ۀ‬஍ଳ

৿ྛ

࣪஍

ਫҬ

มԽͳ͠

มԽ͋Γ

ग़‫[ ཰֬ݱ‬%]

17.70

11.51

61.84

8.36

0.02

0.58

99.23

0.77

ΫϥεॏΈ

0.0

2.16

0.48

2.48

8.54

5.15

0.01

4.87

͖Δɽ
ηϚϯςΟοΫηάϝϯςʔγϣϯɿද 2 ΑΓɼఏҊख๏͸શͯͷධՁࢦඪͰ࠷΋ߴਫ਼౓Ͱ͋Δ͜ͱ͕֬ೝͰ͖Δɽಛʹ U-Net (Seg) ͸ɼఏҊख๏ͷηάϝϯςʔγϣϯϞσ
ϧͱಉҰͷωοτϫʔΫߏ଄Ͱ͋Δ͕ɼࣝผਫ਼౓͕࠷େ໿
0.7 ϙΠϯτ޲্ͨ͠ɽ͜Ε͸ɼλεΫؒͰ৘ใΛ૬‫ޓ‬ར༻͔ͨ͠ΒͰ͋Δͱߟ͑ΒΕΔɽ·ͨɼਤ 3 ΑΓ೶‫ۀ‬஍ଳͳͲͷྖҬͷ޿͍Ϋϥεͱɼ‫ݕ‬ग़͕ࠔ೉ͳ઒ͳͲྖҬ͕ࡉ͍Ϋϥεʹ͓͍ͯ΋ߴਫ਼౓ͳηάϝϯςʔγϣϯ͕Ͱ͖ͯ
͍Δ͜ͱ͕෼͔Δ.

4.4 Ϋϥεෆ‫ʹߧۉ‬ର͢Δ༗ޮੑͷௐࠪਓ޻෺ΫϥεΛ‫ؚ‬Ήύονը૾Λ௥Ճ͢Δ͜ͱʹΑΔਫ਼
౓ൺֱΛߦ͏ɽද 3 ʹɼ௥Ճલͱ௥Ճ‫ͦޙ‬ΕͧΕͷఆྔతධՁΛࣔ͢ɽग़‫཰֬ݱ‬ͷ௿͍ΫϥεΛ‫ؚ‬Ήύονը૾Λ௥Ճ͢Δ͜ͱͰֶश͕҆ఆ͠ɼ΄΅શͯͷΫϥεͰਫ਼౓޲্
Λ֬ೝͨ͠ɽ·ͨɼग़‫཰֬ݱ‬ͷߴ͍೶‫ۀ‬஍ଳͷ‫ࣝޡ‬ผ͕‫ݮ‬গͨ͜͠ͱͰɼ৿ྛΫϥεͷࣝผਫ਼౓͕໿ 35.7 ϙΠϯτ޲্ͨ͠ɽ
ද 3 : ೖྗํ๏มߋʹΑΔൺֱ݁Ռ

4.2 ࣮‫֓ݧ‬ཁHRSCD σʔληοτ͸ɼֶशαϯϓϧ͕গͳ͍ͨΊɼ‫ز‬Կม‫׵‬ɼϊΠζ෇Ճ౳ͷ Data Augmentation Λߦ͍ɼόϦΤʔγϣϯΛ૿Ճ͢ΔɽධՁ࣌͸ɼ௒ߴղ૾౓ը૾Λ
512×512 ϐΫηϧͷը૾ͱͯ͠λΠϧঢ়ʹ੾Γग़ͯ͠ೖྗ
͢ΔɽఆྔతධՁࢦඪͱͯ͠ɼมԽ఺‫ݕ‬ग़ɼηϚϯςΟο
Ϋηάϝϯςʔγϣϯ‫ ʹڞ‬Global Accuracyɼmean IoUɼKappa ܎਺Λ༻͍ΔɽηϚϯςΟοΫηάϝϯςʔγϣϯͰ͸ɼ͞Βʹ Class Accuracy Λ༻͍Δɽ·ͨɼఏҊख๏ͷੑೳൺֱͱͯ͠ɼU-Net ͓Αͼ CNNF-FɼFC-EF-Res Λ
࢖༻͢ΔɽU-Net ͸ɼ୯ҰͷωοτϫʔΫͱͯ͠มԽ఺‫ݕ‬ग़λεΫ U-Net (Diff) ͱηϚϯςΟοΫηάϝϯςʔγϣϯλεΫ U-Net (Seg) ͦΕͧΕͷֶशΛߦ͏ɽ

4.3 ֤λεΫʹ͓͚ΔධՁ݁Ռ֤ख๏Λ࢖༻ͨ࣌͠ͷɼมԽ఺‫ݕ‬ग़ͱηϚϯςΟοΫη
άϝϯςʔγϣϯͷఆྔతධՁΛද 2 ʹɼఆੑతධՁΛਤ
2 ͓Αͼਤ 3 ʹࣔ͢. U-Net (Seg) Ͱ͸ɼ֤ը૾ʹର͢Δηάϝϯςʔγϣϯ݁ՌΛར༻ͨ͠มԽ఺‫ݕ‬ग़Λߦ͓ͬͯΓɼը૾ؒͰΫϥε͕ҟͳΔྖҬΛมԽྖҬͱ͢Δɽ
มԽ఺‫ݕ‬ग़ɿද 2 ΑΓɼఏҊख๏͕ैདྷख๏ͱൺֱ͠ɼmean
IoU ͱ Kappa ܎਺ʹ͓͍ͯߴਫ਼౓Ͱ͋Δ͜ͱ͕֬ೝͰ͖
ΔɽҰํɼU-Net (Seg) Ͱ͸ηάϝϯςʔγϣϯਫ਼౓͸ߴ͍΋ͷͷɼ‫ݕ‬ग़ਫ਼౓͸௿Լͨ͠ɽ͜Ε͸ɼηάϝϯςʔγϣϯ݁Ռʹ͓͍ͯΫϥε‫ڥ‬քͷ‫ࣝޡ‬ผ͕ଟ͋͘ΔͨΊɼมԽྖҬͷ‫ݕޡ‬ग़Λট͍ͨͱߟ͑ΒΕΔɽ·ͨɼਤ 2 ΑΓఏҊख๏͸ಓ࿏ͳͲͷࡉ͍ྖҬͷ‫ݕ‬ग़ՄೳͰ͋Δ͜ͱ͕֬ೝͰ

Semantic Segmentation

Change Detectionਓ޻෺೶‫ۀ‬஍ଳ

৿ྛ

࣪஍

ਫҬ

มԽͳ͠

௥Ճલ

67.51

85.27

37.81

0.0

31.56

99.20

มԽ͋Γ

17.74

௥Ճ‫ޙ‬

68.97

89.88

73.55

0.0

35.01

99.17

19.47

5. ͓ΘΓʹ
ຊ‫Ͱڀݚ‬͸ɼมԽྖҬͷΦϒδΣΫτΫϥεΛਪఆ͢Δ
͜ͱͷͰ͖ΔมԽ఺‫ݕ‬ग़๏ΛఏҊͨ͠ɽ2 ͭͷϞσϧΛ࢖
༻͢Δ͜ͱͰɼߴਫ਼౓ͳมԽྖҬͷ‫ݕ‬ग़Λ࣮‫͠ݱ‬ɼ୯Ұͷ
U-Net ಉ༷ͷηάϝϯςʔγϣϯਫ਼౓Λୡ੒ͨ͠ɽࠓ‫ޙ‬͸ɼग़‫཰֬ݱ‬ͷ௿͍ΫϥεͷߋͳΔਫ਼౓޲্Λ‫ݕ‬౼͢Δɽ

ࢀߟจ‫ݙ‬[1] E. Amin, et al., “Convolutional neural network features
based change detection in satellite images”, First International Workshop on Pattern Recognition, 2016.
[2] R. C. Daudt, et al., “Multitask Learning for Large-scale
Semantic Change Detection”, CVIU, 2019.
[3] O. Ronneberger, et al., “U-Net: Convolutional Networks
for Biomedical Image Segmentation”, MICCAI, 2015.

‫੷ۀڀݚ‬[1] ౵Ҫॣ‫ ޗ‬౳,“ηϚϯςΟοΫηάϝϯςʔγϣϯʹΑΔ௒ߴ
ղ૾౓ը૾͔ΒͷมԽ఺‫ݕ‬ग़”, ಈతը૾ॲཧ࣮ར༻ԽϫʔΫγϣοϓ, 2021.
(ଞ 1 ݅)ҟͳΔղ૾౓ؒͷ੔߹ੑΛߟྀͨ͠‫ݦ‬ஶੑ༧ଌϞσϧʹؔ͢Δ‫ڀݚ‬ࢦಋ‫ڭ‬तɿࢁԼོٛ

TP19010 ੉ඌढ़‫و‬
1. ͸͡Ίʹ
‫ݦ‬ஶੑ༧ଌ͸ɼਓ͕‫ڵ‬ຯɾؔ৺Λ࣋ͭͱߟ͑ΒΕΔྖҬΛώʔτϚοϓʹΑΓද‫͢ݱ‬Δɽ‫ݦ‬ஶੑ༧ଌͷΞϓϩʔνͱͯ͠ɼपғͱ͸ҟͳΔ৭࠼΍Τοδಛ௃Λநग़ͯ͠‫ݦ‬ஶྖҬΛਪఆ͢Δख๏͕͋Δɽ·ͨɼConvolutional Neural
Network (CNN) ͷൃలʹΑΓɼਓͷࢹઢ͔ΒಘΒΕͨ‫ݦ‬ஶੑϚοϓΛֶशɾਪ࿦ʹ༻͍Δߴਫ਼౓ͳ‫ݦ‬ஶੑ༧ଌख๏
΋ఏҊ͞Ε͍ͯΔɽҰํͰɼ‫ݦ‬ஶੑ༧ଌͷԠ༻ઌͱͯࣗ͠
ಈӡసγεςϜΛ૝ఆ͢Δͱɼ༧ଌਫ਼౓͚ͩͰ͸ͳ͘ϝϞϦফඅྔ΍‫ؒ࣌ࢉܭ‬ͷ୹ॖ͕ॏཁͱͳΔɽͦ͜Ͱɼຊ‫ڀݚ‬Ͱ͸ϝϞϦফඅྔΛޮ཰Խͨ͠‫ݦ‬ஶੑ༧ଌϞσϧΛఏҊ͢
Δɽ͞Βʹɼύϥϝʔλͷগͳ͍ϞσϧͰߴਫ਼౓ʹਪఆ͢ΔͨΊʹɼը૾ղ૾౓ຖͷ‫ݦ‬ஶੑͷҰ؏ੑΛߟֶྀͨ͠श
ख๏΋ఏҊ͢Δɽ

2. ‫ݦ‬ஶੑ༧ଌ‫ݦ‬ஶੑ༧ଌͷ୅දతͳख๏ͱͯ͠ɼRARE2012 [1] ͕‫ڍ‬͛ΒΕΔɽRARE2012 Ͱ͸ɼओ੒෼෼ੳʹΑΓΧϥʔνϟϯωϧຖʹന৭ԽΛߦ͏ɽͦͷ‫ޙ‬ɼಘΒΕͨը૾ʹରͯ͠ϚϧνεέʔϧͷΨϘʔϧϑΟϧλͰಛ௃ྔΛநग़͢Δɽ࠷‫ʹޙ‬ɼಘΒΕͨಛ௃ྔͷ֬཰ີ౓Λ‫͠ࢉܭ‬ɼࣗ‫ݾ‬৘ใྔʹ‫ݦ͍ͯͮج‬ஶੑ༧ଌΛߦ͏ɽͨͩ͠ɼRARE2012 Ͱ͸‫ة‬‫ݥ‬༧஌ͳͲਓͷࣄલ஌ࣝʹ΋ͱͮ͘‫ݦ‬ஶੑͷҧ͍ΛߟྀͰ͖ͳ͍ͱ͍͏໰୊͕͋Δɽ·ͨɼਓͷࢹઢ৘ใΛར༻ͨ͠
CNN ʹΑΔߴਫ਼౓ͳ‫ݦ‬ஶੑਪఆख๏͕͋ΔɽSalNet [2] Ͱ͸ɼ৞ΈࠐΈ૚ 3 ૚ɼશ݁߹૚ 2 ૚͔Βߏ੒͞ΕΔωοτϫʔΫΛ༻͍ͯ‫ݦ‬ஶੑϚοϓΛग़ྗ͢ΔɽωοτϫʔΫͷֶशʹ͸ਓͷࢹઢσʔλΛ΋ͱʹͨ͠ϚοϓΛ‫ͯ͠ͱࢣڭ‬༻͍Δɼ·ͨɼEML-Net [3] Ͱ͸ɼҟͳΔσʔληοτͰࣄલֶशͨ͠ෳ਺ωοτϫʔΫͷࢹ֮ಛ௃Λ౷߹͢Δ͜ͱ
Ͱߴਫ਼౓ͳ༧ଌ͕ՄೳͰ͋Δɽ͔͠͠ɼEML-Net ͸ɼେ‫ن‬໛ͳωοτϫʔΫΛར༻ͯ͠ಛ௃நग़Λߦ͏ͨΊɼϞσϧͷύϥϝʔλ਺͕๲େͱͳΓɼ‫͔͔͕ؒ࣌ࢉܭ‬Δ໰୊͕͋Δɽ

3. ఏҊख๏
ຊ‫Ͱڀݚ‬͸ɼϝϞϦফඅྔ‫ؒ࣌ࢉܭͼٴ‬ͷ୹ॖͱ༧ଌਫ਼
౓ͷ޲্Λཱ྆͢Δ‫ݦ‬ஶੑ༧ଌख๏ΛఏҊ͢Δɽ·ͣɼϝϞϦফඅྔͱ‫ؒ࣌ࢉܭ‬ͷ୹ॖͷͨΊʹɼMobileNetV2 ͱ
Mixed Depthwise Convolution Λར༻ͯ͠ωοτϫʔΫΛߏங͢Δɽ͜ΕʹΑΓɼύϥϝʔλ਺Λ࡟‫͖Ͱݮ‬Δɽ࣍
ʹɼ༧ଌਫ਼౓Λ޲্ͤ͞ΔͨΊʹɼਓͷࢹ֮తಛੑʹ‫ͮج‬͍ͨղ૾౓ຖͷࢹ֮తಛ௃Λޮ཰Αֶ͘श͢Δଛࣦؔ਺ΛఏҊ͢Δɽը૾ղ૾౓ͱ‫ݦ‬ஶੑϚοϓͷؔ܎ੑʹ͍ͭͯɼਓͷࢹઢΛऩू͠ௐࠪͨ͠‫͕͋ڀݚ‬Δ [4]ɽ͜ΕʹΑΔͱɼղ૾౓Λ32 ෼ͷ 1 ·ͰϦαΠζͨ͠ը૾ͷ‫ݦ‬ஶੑϚοϓ͸‫ݩ‬ͷղ૾౓ͷ‫ݦ‬ஶੑϚοϓͱඇৗʹ૬͕ؔߴ͍͜ͱ͕֬ೝ͞Ε͍ͯΔɽ͜ͷ஌‫ݟ‬Λ‫͔͠׆‬ɼ‫ݩ‬ͷղ૾౓͔Β༧ଌ͞ΕΔ‫ݦ‬ஶੑϚοϓͱ௿ղ૾౓Խͨ͠ը૾͔Β༧ଌ͞ΕΔ‫ݦ‬ஶੑϚοϓͷ੔߹ੑΛଛࣦͱͯ͠‫ٻ‬ΊΔɽ͜ΕΛఆࣜԽ͢Δͱࣜ (1)ͷΑ͏ʹఆٛͰ͖Δɽ

Lres = BCE(S GT , S Pb ) +N!

BCE(S Pb , S P2i )

(1)

i=1

ୈҰ߲͸ɼόΠφϦΫϩεΤϯτϩϐʔΛ༻͍ͨਅ஋ͱ
‫ݩ‬ͷղ૾౓͔Β༧ଌ͞ΕΔ‫ݦ‬ஶੑϚοϓͷଛࣦͰ͋Δɽୈೋ߲͸ɼ‫ݩ‬ͷղ૾౓͔Β༧ଌ͞ΕΔ‫ݦ‬ஶੑϚοϓͱ௿ղ૾౓Խͨ͠ը૾͔Β༧ଌ͞ΕΔ‫ݦ‬ஶੑϚοϓ͔ΒͷଛࣦͰ͋Δɽ͜͜ͰɼBCE ͸όΠφϦΫϩεΤϯτϩϐʔɼS GT͸‫ݦ‬ஶੑϚοϓͷਅ஋ɼS Pb ͸‫ݩ‬ͷղ૾౓͔Β༧ଌ͞Εͨ‫ݦ‬ஶੑϚοϓɼS P2i ͸ղ૾౓Λ 2i ෼ͷ 1 ʹͨ࣌͠ͷਪ࿦
݁ՌɼN ͸μ΢ϯαϯϓϧ͢Δճ਺Ͱ͋Δɽਤ 2 ʹࣔ͢Α͏ʹɼ௿ղ૾౓ͷը૾͸‫ݩ‬ͷղ૾౓ͷը૾Λ 2i ෼ͷ 1 ʹμ΢ϯαϯϓϦϯά‫ޙ‬ɼόΠϦχΞิؒʹΑΓ‫ݩ‬ͷղ૾౓΁ΞοϓαϯϓϦϯάͨ͠ը૾ͱ͠ɼϞσϧʹೖྗ͢Δɽ
32

64

64

32

128

IRL
BN

1
32

ReLU6

×3

64
128

256

256

128

!"#(% &01 ,% &' )

512

IRL
BN
ReLU6

×4

High resolution



1/2-

% &'

% ./



!"#(% &' , % &)* )Weight share

32

64

64

32

128

IRL

ఏҊख๏Ͱ༻͍ΔωοτϫʔΫߏ଄Λਤ 1 ʹࣔ͢ɽ͸͡ΊʹɼMobileNetV2 Λಛ௃நग़‫ͯ͠ͱث‬ը૾͔Βಛ௃Λಘ
Δɽ͜ͷ࣌ɼMobileNetV2 ͸ IRLɼBatch Normalization(BN)ɼReLU6 Λ૊Έ߹Θͤͨ‫ ܭ‬7 ૚ͷߏ੒Ͱ͋Γɼલ൒
ͷ 3 ૚ͱ‫ޙ‬൒ͷ 4 ૚͔Βಛ௃ΛಘΔɽ

BN

1
32

ReLU6

×3

64
128

256

256

128

512

IRL
BN
ReLU6

Low resolution

×4

&

% )*

ਤ 2 : ֤ղ૾౓ؒͷ੔߹ੑΛߟྀͨ͠Ұ؏ੑଛࣦ

MixConv

Input
3 3

32

64

5 5

64

4. ධՁ࣮‫ݧ‬

7 7ఏҊख๏ͷ༗ޮੑΛࣔͨ͢Ίʹɼ੩ࢭըʹ͓͚Δਓͷࢹ
ઢσʔλ͔ΒಘΒΕͨ‫ݦ‬ஶੑΛऩूͨ͠σʔληοτΛ༻͍ͯɼධՁ࣮‫ݧ‬Λߦ͏ɽ

32

128

IRL
BN

1
32

ReLU6

×3
256

256
512

IRL

4.1 ࣮‫֓ݧ‬ཁ

160
128

Output

BN

GT

ReLU6

High feature

3.2 ଛࣦؔ਺



3.1 ωοτϫʔΫߏ଄

Low featureғͱ͸ҟͳΔ৭৘ใ΍ࣝผʹ༗༻ͳΤοδͳͲͷ୯७ͳࢹ֮৘ใͱͳΔɽ·ͨɼ‫ޙ‬൒ͷಛ௃͸ɼਂ͍૚͔ΒಘΒΕΔಛ
༗ͷಛ௃Ͱ͋Δ෺ମͳͲͷେҬతͳࢹ֮৘ใͱͳΔɽ࣍ʹɼಘΒΕͨಛ௃Λ Refinement layer ʹೖྗ͢ΔɽRefinement
layer ͸ಘΒΕͨಛ௃Λ໌ྎԽ͢ΔϞδϡʔϧͱͳ͓ͬͯΓɼೖग़ྗͷղ૾౓Λมߋ͠ͳ͍ Encoder-Decoder ߏ଄͔Βߏ੒͞Ε͍ͯΔɽ͞ΒʹɼRefinement layer ʹ͸༷ʑͳड༰໺Λ 1 ౓ͷ৞ΈࠐΈ૚ʹू໿Λߦͬͨ Mixed-depthwise
Convolution (MixConv) Λ༻͍Δ͜ͱͰɼԋࢉྔΛ཈͑ͭͭ΋ϩόετͳಛ௃ͷநग़Λߦ͏ɽ࠷‫ʹޙ‬ɼRefinement
layer ͔ΒಘΒΕͨ̎ͭͷಛ௃Λ౷߹͠ɼDeconvolution ૚Λ༻͍ͯ‫ݦ‬ஶੑϚοϓͷਪ࿦Λߦ͏ɽ

×4
Feature Extractor(MobileNetv2)

Refinement layer

Decode layerਤ 1 : ఏҊख๏ͷωοτϫʔΫߏ଄લ൒ͷಛ௃͸ɼઙ͍૚͔ΒಘΒΕΔಛ༗ͷಛ௃Ͱ͋ΔपධՁσʔλʹ͸ɼSALICON ‫ ͼٴ‬CAT2000 Λ༻͍Δɽ
SALICON ͸ɼ༷ʑͳࣗવը૾͔Βߏ੒͞ΕΔ MS-COCOσʔληοτ͔Β 20,000 ຕΛൈਮ͠ɼ໿ 60 ໊ͷඃ‫͔ऀݧ‬Β‫ݦ‬ஶੑΛ֫ಘͯ͠෇༩ͨ͠΋ͷͰ͋Δɽ·ͨɼCAT2000
͸ 120 ໊ͷඃ‫ऀݧ‬Λର৅ʹɼອըɼΞʔτɼΦϒδΣΫ
τɼ௿ղ૾౓ը૾ɼ԰಺ɼ԰֎ɼϥϯμϜͳը૾ɼઢըͱ
͍ͬͨҟͳΔλΠϓͷγʔϯΛΧόʔ͠ɼ20 ͷΧςΰϦ
͔Βߏ੒͞Ε͍ͯΔɽը૾ͷೖྗαΠζ͸ 640 ʷ 480 Ͱද 1 : SALICONɼ‫ ͼٴ‬CAT2000 σʔληοτʹΑΔఆྔతධՁ݁Ռ

EML-Net[1]Vanilla
Vanilla+R
Vanilla+R+Lres

Params47.08M
4.72M
5.75M
5.75M

SIM ˢ
0.765
0.686
0.714
0.742

SALICON
CC ˢ AUC ˢ
0.878
0.864
0.779
0.844
0.811
0.850
0.847
0.861

NSS ˢ
1.987
1.577
1.673
1.762

KL ˣ
0.520
0.393
0.338
0.282

Params47.08M
4.72M
5.75M
5.75M

SIM ˢ
0.782
0.740
0.757
0.776

CAT2000
CC ˢ AUC ˢ
0.885
0.866
0.833
0.858
0.842
0.860
0.851
0.866

NSS ˢ
2.060
1.731
1.799
1.875

KL ˣ
0.298
0.321
0.311
0.299

High

Low
GT

VanillaVanilla+RVanilla+R+!"#$

EML-Netਤ 3 : ֤ख๏ʹΑΔ‫ݦ‬ஶੑϚοϓͷਪఆྫ
͋ΔɽSAICON ͸ֶश༻ʹ 10,000 ຕɼධՁ༻ʹ 5,000 ຕΛɼCAT2000 ͸ֶश༻ʹ 1,600 ຕɼධՁ༻ʹ 400 ຕΛ༻͍Δɽ·ͨɼֶशϞσϧͷ൚Խੑೳͷ‫͔఺؍‬Βલॲཧͱ͠
ͯೖྗը૾Λ [0,1] ΁ਖ਼‫ن‬Խɼ‫ࠨͼٴ‬ӈ൓సͷσʔλ֦ுΛߦ͍ͬͯΔɽධՁʹར༻͢Δैདྷख๏ͱϞσϧΛҎԼͷΑ͏ʹఆٛ͢Δɽ
EML − Net : EML-Net ͸ɼෳ਺ͷࣄલֶशࡁΈϞσϧΛ‫ֶͨ͠༻׆‬शΛߦ͏͜ͱͰɼࣄલʹΤϯίʔυ͞Εͨࣝผʹ༗ޮͳಛ௃ΛޮՌతʹར༻͢Δख๏Ͱ͋Δɽ
Vanilla : MobileNetV2 ͷσίʔυ෦Λશ݁߹૚͔Β Deconvoliution ૚Λ 3 ૚ʹมߋ͢Δ͜ͱͰɼ‫ݦ‬ஶੑ༧ଌ༻ʹωοτϫʔΫΛมߋͨ͠ϞσϧͰ͋Δɽ
Vanilla + R(Ours) : Refinement layer ‫ ͼٴ‬MixConv Λ‫֫ͯ͠༻׆‬ಘͨ͠ಛ௃Λ౷߹ͯ͠ Deconvolution ૚΁ೖྗ͢ΔϞσϧͰ͋Δɽ
Vanilla + R + Lres (Ours) : Vanilla+R ͷϞσϧʹରͯ͠ɼ֤ղ૾౓ͷ੔߹ੑΛߟྀͨ͠ଛࣦؔ਺ Lres Λಋೖͨ͠ϞσϧͰ͋Δɽ

4.2 ϕʔεϥΠϯͱͷఆྔతɾఆੑతͳൺֱ
‫ݦ‬ஶੑ༧ଌʹ͓͚ΔఆྔతධՁࢦඪͱͯ͠ɼSIM(Similarity)ɼCC(Correlation coefficient)ɼAUC(Area Under
Curve)ɼNSS(Normalized Scanpath Saliency)ɼKL(KLdivergence) ͕͋Δɽ͜ΕΒͷධՁࢦඪΛ༻͍ͯɼSALICON σʔληοτͱ CAT2000 σʔληοτʹ͓͚Δ֤
ਫ਼౓ͷൺֱ݁Ռͱύϥϝʔλ਺Λද 1 ʹࣔ͢ɽఏҊख๏͸ɼ੔߹ੑΛอͭଛࣦؔ਺ Lres ͱ Refinement layer ʹΑΓɼैདྷख๏ͱൺֱͯ͠΄΅ಉ౳ͷਫ਼౓Ͱ͋Δ͜ͱ͕Θ͔Δɽ
·ͨɼIRL ͱ MixConv Λར༻͢Δ͜ͱʹΑΓɼύϥϝʔλ਺Λ 87.7%࡟‫͍͖ͯͰݮ‬Δ͜ͱ͕֬ೝͰ͖Δɽ࣍ʹɼਤ 3 ʹ֤ख๏ʹΑΔ‫ݦ‬ஶੑϚοϓͷਪఆྫΛࣔ͢ɽ
ਤ 3 ΑΓɼVanilla Ͱ͸‫ؠ‬΍໦ࡐͰฤ·ΕͨҜࢠͳͲΤοδ౳ͷಛ௃͕ີू͢ΔҐஔʹ‫ݦͯͬޡ‬ஶੑ͕‫ݱ‬Ε͍ͯΔɽ
ҰํɼఏҊख๏͸ Refinement layer ͱ Lres ͷಋೖʹΑΓ‫ޡ‬Γͷগͳ͍‫ݦ‬ஶੑϚοϓͱͳ͍ͬͯΔɽ

4.3 ଛࣦؔ਺ͷ༗ޮੑͷௐࠪ
ਤ 4 ʹɼҟͳΔղ૾౓ؒͷը૾Λೖྗͱͨ͠ͱ͖ͷ CC
ʹΑΔਫ਼౓มԽΛࣔ͢ɽͳ͓ɼ‫ݩ‬ͷղ૾౓Λԣ࣠ͷ 1 ͱ͠ɼӈ΁ҠΔʹͭΕͯղ૾౓͕ 2 ෼ͷ 1 ͣͭ௿Լ͍ͯ͠Δɽਤ
4 ͔ΒɼҰ؏ੑଛࣦΛ༻͍͍ͯͳ͍ Vanilla+R (Without
Lres ) ͱൺ΂ɼVanilla+R+Lres (With Lres ) Ͱ͸௿ղ૾౓Ͱ΋ਫ਼౓͸௿Լ͠ͳ͍ࣄ͕֬ೝͰ͖ΔɽਓؒͷࢹઢͰ͸ɼ32 ෼ͷ 1 ·Ͱͷղ૾౓Ͱ͋Ε͹Ұఆͷ‫ݦ‬ஶੑϚοϓ͕ಘΒΕΔ͜ͱ͔Βɼ֤ղ૾౓ؒͷ੔߹ੑΛߟྀͨ͠Ұ؏ੑଛ

ਤ 4 : ෳ਺ղ૾౓ʹର͢ΔఏҊख๏ͷਫ਼౓ൺֱࣦΛར༻͢Δ͜ͱͰɼਓؒʹ͍ۙࢹ֮తಛ௃ΛֶशͰ͖ͯ
͍Δͱ͍͑Δɽ

5. ͓ΘΓʹ
ຊ‫Ͱڀݚ‬͸ɼ‫ݦ‬ஶੑ༧ଌʹ͓͍ͯ MobileNetV2 ΍ MixConv Λར༻ͨ͠ωοτϫʔΫͷઃ‫ʹܭ‬ΑΓύϥϝʔλ਺Λ໿ 87.7%࡟‫ͨ͠ݮ‬ɽ·ͨɼ֤ղ૾౓ؒͷ‫ݦ‬ஶੑͷ੔߹ੑΛอͭଛࣦؔ਺ Lres ͷఏҊʹΑΓɼLres Λར༻͠ͳ͍৔
߹ͱൺ΂ਫ਼౓͕޲্͠ɼैདྷख๏ͱൺ΂ͯύϥϝʔλ਺Λ࡟‫ʹͨ͠ݮ‬΋ؔΘΒͣɼ΄΅ಉ౳ͷਫ਼౓ͱͳΔ͜ͱΛ֬ೝͨ͠ɽࠓ‫ޙ‬͸ɼߋͳΔਫ਼౓޲্Λૂ͏ͨΊʹΞϯαϯϒϧ
Ϟσϧ΍ AutoML ʹΑΔωοτϫʔΫߏ଄ͷ࠷దԽΛߦ͏༧ఆͰ͋Δɽࢀߟจ‫ݙ‬[1] N. Riche, et al., “A multi-scale rarity-based saliency detection with its comparative statistical analysis”, SPIC,
2013.
[2] J. Pan, et al., “Shallow and Deep Convolutional Networks for Saliency Prediction”, CVPR, 2016.
[3] S. Jia, et al., “EML-NET:An Expandable Multi-Layer
NETwork for Saliency Prediction”, IVC, 2020.
[4] T. Judd, et al., “Fixations on low-resolution images.”,
JOV, 2011.

‫੷ۀڀݚ‬[1] ੉ඌढ़‫ و‬౳, “FlowNetC Λಋೖͨ͠ D&T ʹ͓͚Δ෺ମ‫ݕ‬ग़ͷߴਫ਼౓Խ”, ը૾ηϯγϯάγϯϙδ΢Ϝ, 2019.
[2] T. Seo, et al, “Video Object Detection and Tracking
based on Angle Consistency between Motion and Flow”,
IV, 2020.
(ଞ ֶձൃද 1 ݅)ࢹઢ৘ใΛ‫ͨ͠༻׆‬Ұ؏ֶशϕʔεͷࣗಈӡస੍‫ؔ͢ʹޚ‬Δ‫ڀݚ‬ࢦಋ‫ڭ‬तɿࢁԼོٛ

TP19023 ৿‫ܒ‬հ
1. ͸͡Ίʹ

視線推定モデル2. Ұ؏ֶशϕʔεͷࣗಈӡస੍‫ޚ‬୅දతͳҰ؏ֶशϕʔεͷࣗಈӡసख๏ͱͯ͠ɼBojarskiΒͷ‫[ ڀݚ‬1] ͕͋Δɽ͜ͷ‫Ͱڀݚ‬͸ɼ5 ૚ͷ৞ΈࠐΈ૚ͱ 4 ૚ͷશ݁߹૚ͷ CNN ʹը૾Λೖྗ͠ɼεςΞϦϯά஋Λਪఆ͢Δɽ·ͨɼConditional Imitation Learning RS (CILRS)[2] ͸ɼResNet Λಛ௃நग़෦ʹಋೖ͠ɼϚϧνλεΫͱͯ͠ं྆଎౓ਪఆΛ௥Ճ͢Δ͜ͱʹΑΓɼӡసੑೳΛվળ͠
͍ͯΔɽCILRS Ͱ͸ɼશ݁߹૚Ͱߏ੒ͨ͠ग़ྗ෦Λ௚ਐ΍ӈࠨંͱ͍ͬͨίϚϯυͷ਺͚ͩ༻ҙ͍ͯ͠ΔɽίϚϯυʹԠͯ͡ग़ྗ෦Λ੾Γସ͑Δ͜ͱͰɼҰ؏ֶशϕʔεͷࣗಈӡసʹ͓͍ͯࠔ೉Ͱ͋ͬͨɼަࠩ఺Ͱͷߴਫ਼౓ͳ੍‫ޚ‬Λ࣮‫͍ͯ͠ݱ‬Δɽ͔͠͠ɼӡస੍‫ྖ͍ͳ܎ؔʹޚ‬Ҭʹର͢
Δಛ௃Λநग़͢Δ͜ͱͰɼ੍‫ޚ‬஋ͷਖ਼͍͠ਪ࿦ʹࣦഊ͢Δ
ͱ͍͏໰୊͕ߟ͑ΒΕΔɽ

3. ఏҊख๏
ຊ‫Ͱڀݚ‬͸ɼਓͷࢹઢ৘ใΛར༻͢Δ͜ͱͰɼӡసʹඞཁͳྖҬΛॏࢹͰ͖Δӡస੍‫ޚ‬ϞσϧΛఏҊ͢ΔɽఏҊ͢ΔϞσϧͷωοτϫʔΫΛਤ 1 ʹࣔ͢ɽ

3.1 ࢹઢਪఆϞσϧࣗಈӡస࣌ʹɼਓ͕஫ࢹ͢ΔҐஔΛৗʹ‫ܭ‬ଌ͢Δͷ͸ਓ΁ͷෛ୲͕େ͖͍ɽͦ͜Ͱɼਓͷ஫ࢹྖҬΛද͢ࢹઢϚοϓΛਪఆ͢ΔࢹઢਪఆϞσϧΛ࣮‫͠ݱ‬ɼਪఆͨ͠ࢹઢ৘ใ
Λӡస੍‫ޚ‬Ϟσϧʹར༻͢Δɽ͜ΕʹΑΓɼਓͷࢹઢ৘ใΛ༻͍Δ͜ͱͳ͘ɼӡస੍‫ޚ‬Ϟσϧͷֶश͓ΑͼධՁ͕Մ
ೳͱͳΔɽࢹઢਪఆϞσϧͷωοτϫʔΫ͸Τϯίʔμ͓ΑͼσίʔμͰߏ੒͢ΔɽΤϯίʔμ͸ VGG-16 ͷ৞ΈࠐΈ૚Λɼσίʔμ͸ 4 ૚ͷ deconvolution ૚Λ༻͍Δɽ
ަࠩ఺ͷ৔໘Ͱ͸ɼਐߦํ޲ʹΑͬͯ஫ࢹ͢ΔՕॴ͕େ͖
͘ҟͳΔɽͦͷͨΊɼӡసࢦࣔͰ͋ΔίϚϯυ c ΛωοτϫʔΫʹ௥Ճ͢Δ͜ͱͰɼӡసࢦࣔʹ߹ΘͤͨࢹઢਪఆΛ
ֶश͢ΔɽίϚϯυ͸ɼΤϯίʔμ͔Β֫ಘ͞Εͨಛ௃ϚοϓͱಉαΠζʹϦαΠζ͠ɼνϟϯωϧํ޲ʹ݁߹ͯ͠௥
Ճ͢ΔɽࢹઢਪఆϞσϧͷֶशʹ͸ɼࣜ (1) ʹࣔ͢ Binary
Cross Entropy (BCE) Loss Λ༻͍Δɽ͜͜ͰɼGɼĜ ͸ͦΕͧΕࢹઢϚοϓͷਅ஋ͱ༧ଌ஋Ͱ͋Δɽ
W

Lg =Deconv.

VGG-16

視線マップ Ĝ

コマンド cLinearLinear左折

LinearResNet34

右折Linear
計測値 (速度) m

運転制御モデル

速度 vt

車線追従

4

Res.

D.S.

Blocks

3

Res.

D.S.

Blocks

Res.

Conv. Layer

D.S.

Blocks 2

D.S.

入力画像

Res. Blocks 1ࣗಈӡస੍‫ޚ‬͸ɼपล‫ڥ؀‬ͷೝࣝɼ‫ܦ‬࿏‫ܭ‬ըɼं੍྆‫ޚ‬Λॱ࣍ߦ͏Ξϓϩʔν͕༻͍ΒΕΔɽҰํɼConvolutional
Neural Network (CNN) ͷൃలʹΑΓɼ৽ͨͳࣗಈӡస੍‫ͯ͠ͱޚ‬ɼंࡌΧϝϥը૾͔Β੍‫ޚ‬஋Λ௚઀ਪఆ͢ΔҰ؏
ֶशϕʔεͷख๏͕ఏҊ͞Ε͍ͯΔ [1]ɽຊख๏͸ɼਓ͕࣮ؒࡍʹӡసͨ͠ࡍͷ৘ใΛ‫ࢣڭ‬σʔλͱͯ͠໛฿ֶशΛߦ͏ɽೖྗʹ͸ओʹը૾͕༻͍ΒΕɼը૾ʹ͸എ‫Ͳͳܠ‬ͷӡస੍‫ʹޚ‬ແؔ܎ͳྖҬ͕ଟ͘‫·ؚ‬ΕΔ͜ͱ͕͋Δɽ͜ͷ࣌ɼCNN ͕ͦΕΒͷෆཁͳྖҬΛ஫ࢹ͢Δͱɼਖ਼੍͍͠
‫ޚ‬஋ΛਪఆͰ͖ͳ͍͜ͱ͕ߟ͑ΒΕΔɽਓؒͷࢹ֮γεςϜ͸ɼࢹઢҠಈʹΑͬͯॏཁͱͳΔ৘ใΛ֫ಘͯ͠ɼࢹ֮
తʹೝࣝΛߦ͏͜ͱ͕஌ΒΕ͍ͯΔɽͦ͜Ͱຊ‫Ͱڀݚ‬͸ɼࢹઢ৘ใΛӡస੍‫ޚ‬Ϟσϧʹར༻ͨ͠Ұ؏ֶशʹΑΔࣗಈӡసΛ࣮‫͢ݱ‬Δɽ

Linear直進

制御値
ステアリングスロットル
ブレーキ

Linearコマンド cਤ 1 : ఏҊख๏ͷωοτϫʔΫߏ଄ಛ௃நग़෦Ͱ͸ɼԼҐ૚ͰΤοδͳͲͷৄࡉͳಛ௃Λநग़͠ɼ্Ґ૚ͰΑΓେ‫ہ‬తͳۭؒಛ௃Λ֫ಘ͢Δɽຊख๏Ͱ
͸ɼࢹઢ৘ใΛਂ͞ͷҟͳΔ 4 Օॴʹ௥Ճ͢Δɽ͜ΕʹΑΓɼεέʔϧͷҟͳΔಛ௃͝ͱʹॏཁͳྖҬΛ‫ڧ‬ௐͰ͖Δɽӡస੍‫ޚ‬Ϟσϧͷग़ྗ෦͸ίϚϯυ͝ͱʹ༻ҙ͢ΔɽίϚϯυ c ʹରԠͨ͠ग़ྗ෦Λબ୒ͯ͠ɼ੍‫ޚ‬஋Λग़ྗ͢Δɽֶश࣌ʹ͸ɼӡస੍‫ޚ‬Ϟσϧͷ֤ग़ྗʹରͯ͠ L1 Loss Λଛࣦؔ਺ʹ༻͍Δɽ͜͜Ͱɼӡస੍‫ޚ‬Ϟσϧͷग़ྗ͸ɼं྆
଎౓ͱεςΞϦϯάɼεϩοτϧɼϒϨʔΩͷ֤஋Ͱ͋Δɽ

4. ධՁ࣮‫ݧ‬ఏҊख๏ͷ༗ޮੑΛௐࠪ͢ΔͨΊɼࢹઢ৘ใͷ༗ແʹΑ
Δӡసਫ਼౓ͷൺֱΛߦ͏ɽ

4.1 σʔληοτຊ࣮‫Ͱݧ‬͸ɼࣗಈӡసγϛϡϨʔλͰ͋Δ CARLA[3]ʹͯ࡞੒͞Εͨ CARLA100 σʔληοτΛӡస੍‫ޚ‬Ϟσϧͷֶशʹ༻͍ΔɽCARLA100 σʔληοτ͸ɼं྆ͷҐஔ΍৴߸‫ػ‬ͷঢ়ଶͳͲͷ಺෦৘ใ͔ΒࣗಈӡసΛߦ͏ΤΩεύʔτΤʔδΣϯτΛ༻͍ͯɼ100 ࣌ؒʹ͓ΑͿӡసσʔλΛγϛϡϨʔλ্Ͱࣗಈతʹऩूͨ͠σʔληοτͰ͋ΔɽࢹઢਪఆϞσϧͷֶशʹ͸ɼCARLA100 σʔλ͔ΒϥϯμϜͰબ୒ͨ͠ 2,000 ຕͷσʔλʹରͯ͠ɼࢹઢ৘ใΛ෇༩ͨ͠σʔληοτΛ༻͍Δɽ1 ϑϨʔϜຖʹे෼ͳࢹઢ৘ใ͕‫·ؚ‬ΕΔΑ͏ʹ͢ΔͨΊɼ੩ࢭըʹର͢Δࢹઢ৘ใΛ‫ه‬࿥͢Δɽࢹઢ৘ใͷऔಘʹ͸ Tobii Pro X3 130 Λ
࢖༻͠ɼ1 ຕͷ੩ࢭըΛσΟεϓϨΠʹ 5 ඵؒදࣔ͢Δ͜ͱͰࢹઢσʔλΛऩूͨ͠ɽຊσʔληοτͷࢹઢσʔλͷྫΛਤ 2 ʹࣔ͢ɽ(a) 動的物体(b) 信号機(c) 左折(d) 右折H

1 !!
−Gi,j log Ĝi,j −(1−Gi,j ) log(1− Ĝi,j )WH i j(1)

3.2 ӡస੍‫ޚ‬Ϟσϧӡస੍‫ޚ‬Ϟσϧʹ͸ CILRS Λϕʔεʹ༻͍Δɽࢹઢ৘ใΛߟྀ͢ΔͨΊɼಛ௃நग़෦Λߏ੒͢Δ Res. Block ຖͷಛ௃Ϛοϓ Fi (x) ʹࢹઢϚοϓ Gi Λըૉຖʹ৐ࢉ͢Δɽ
Ĝi ͸ Ĝ Λ i ൪໨ͷ Res. Block ʹ͓͚Δಛ௃Ϛοϓͷղ૾౓ʹμ΢ϯαϯϓϦϯάͨ͠΋ͷͰ͋Δɽͦͯ͠ɼ஫ࢹ
ྖҬҎ֎ͷಛ௃ྔ͕ফࣦ͢ΔͷΛ๷͙ͨΊɼࣜ (2) ʹࣔ͢Α͏ʹ৐ࢉલͷಛ௃ϚοϓΛՃࢉ͢Δɽ

Fi! (x) = Fi (x) ⊗ Ĝi + Fi (x)(2)ਤ 2 : ‫ه‬࿥ͨ͠ࢹઢ৘ใͷྫ

4.2 ࣮‫֓ݧ‬ཁࢹઢਪఆϞσϧͷೖྗ͸ 400×176 ʹϦαΠζͨ͠ RGB
ը૾Λ༻͍Δɽ࠷దԽख๏ʹ͸ MomentumSGD Λ༻͍ɼֶश཰Λ 0.03 ͱ͢Δɽӡస੍‫ޚ‬Ϟσϧ͸ɼCARLA100 σʔληοτ͔Β 10 ࣌ؒͷαϒηοτΛֶशʹ༻͍Δɽ͜͜Ͱɼೖྗ͸ 200 × 88
ͷ RGB ը૾Λ༻͍Δɽ࠷దԽख๏ʹ͸ Adam Λ࢖༻͠ɼֶश཰Λ 0.0002 ͱ͢Δɽֶश͸ 2 ஈ֊ʹ෼͚ͯߦ͏ɽ͸͡ΊʹࢹઢਪఆϞσϧΛ
ֶश͢Δɽͦͯ͠ɼࢹઢਪఆϞσϧͷॏΈΛ‫ݻ‬ఆͯ͠ӡస

入力画像

出力

ද 1 : NoCrash Benchmark ੒ޭ཰ͷൺֱ݁Ռ [%]࣮‫ݧ‬৚݅

Training
condition
コマンド：車線追従，steer：0.02，throttle：0.00(a) 歩行者

New weather

New town

コマンド：右折，steer：-0.27，throttle：0.00

New town &weather(b) 信号機ަ௨৚݅
Empty
Regular
Dense
Empty
Regular
Dense
Empty
Regular
Dense
Empty
Regular
Dense

ฏ‫ۉ‬

CILRS
92
63
15
92
64
8
54
29
8
72
44
8
45.8

Ours
95
77
25
96
60
28
66
46
12
66
42
12
52.1

ද 2 : িಥࣄ‫ʹނ‬ΑΔΤϐιʔυࣦഊ཰ͷൺֱ݁Ռ [%]࣮‫ݧ‬৚݅

コマンド：車線追従，steer：-0.02，throttle：0.02

Training
condition(c) カーブަ௨৚݅Regular
Dense
Regular

New weather
Dense
Regular

コマンド：右折，steer：0.01，throttle：0.61(d) 交差点

ਤ 3 : ࢹઢϚοϓਪఆྫ : steer ͸ෛ਺Ͱࠨɼਖ਼਺ͰӈͷεςΞϦϯάૢ࡞Λද͢. throttle ͸ɼਖ਼਺ͰΞΫηϧૢ࡞Λද͢ɽ
੍‫ޚ‬ϞσϧΛֶश͢ΔɽධՁ࣮‫Ͱݧ‬͸ɼγϛϡϨʔλ‫ ͯ͠ͱڥ؀‬CARLA Λ༻͍ͨ૸ߦ࣮‫ݧ‬Λߦ͏ɽධՁࢦඪʹ͸ɼNoCrash Benchmark[2] Λ༻͍ΔɽNoCrash Benchmark Ͱ͸ɼCARLA ্ͰҟͳΔλεΫΛ࣮ߦ͠ɼࣗಈӡసͷੑೳΛධՁ͢Δɽ૸
ߦ͢ΔϚοϓ͸ɼֶश‫͋Ͱڥ؀‬Δ Town01 ͱະ஌‫ڥ؀‬ͷ
Town02 Ͱ͋ΔɽλεΫ͸ɼಈత෺ମͷ਺ʹΑΓ Empty
townɼRegular trafficɼDense traffic ͷ 3 छྨʹ෼͚ΒΕɼҟͳΔ‫ڥ؀‬΍ަ௨ঢ়‫͚͓ʹگ‬ΔӡసੑೳΛධՁ͢Δ͜ͱ͕
ՄೳͰ͋Δɽࣄ‫ൃނ‬ੜ࣌·ͨ͸੍‫ؒ࣌ݶ‬Λ௒͑ͨ৔߹ɼΤϐιʔυ͕ࣦഊͱͳΓɼ໨త஍΁౸ୡ͢Δ͜ͱͰΤϐιʔυ੒ޭͱͳΔɽ

4.3 ࢹઢਪఆͷఆੑతධՁ
ࢹઢਪఆ݁ՌͷྫΛਤ 3 ʹࣔ͢ɽਤ 3(a) ͷาߦऀ͕ԣ
அ͢ΔγʔϯͰ͸ɼিಥͷՄೳੑ͕ߴ͍าߦऀʹࢹઢϚοϓ͕൓Ԡ͠ɼ‫ݮ‬଎Λߦ͍ͬͯΔɽਤ 3(b) Ͱ͸ɼલํͷ੺৴߸ʹ൓Ԡ͠ɼӡసʹඞཁͳ෺ମΛ஫ࢹͰ͖͍ͯΔɽਤ 3(c)ͷΑ͏ͳΧʔϒʹ͓͍ͯɼΧʔϒͷߦ͖ઌʹ൓Ԡ͓ͯ͠Γɼਤ 3(d) ͷަࠩ఺ʹ͓͍ͯ͸ɼަࠩ఺ͷӈํ޲ʹ൓Ԡͯ͠
͍Δɽ͜ͷ݁Ռ͔Βɼӡసࢦࣔʹ߹Θͤͨਐߦํ޲Λ஫ࢹͰ͖͍ͯΔ͜ͱ͕Θ͔Δɽ

4.4 ӡసੑೳධՁ
ද 1 ʹ૸ߦ࣮‫ݧ‬ͷ੒ޭ཰Λࣔ͢ɽද 1 ΑΓɼ΄ͱΜͲͷλεΫ‫ڥ؀ͼٴ‬৚݅ʹ͓͍ͯ CILRS ΑΓ੒ޭ཰͕ߴ͍͜
ͱ͕֬ೝͰ͖Δɽ͜ΕʹΑΓɼࢹઢ৘ใ͕ӡసਫ਼౓޲্ʹߩ‫͍ͯ͠ݙ‬Δ͜ͱ͕Θ͔Δɽ·ͨɼਫ਼౓޲্͸ɼͲͷλεΫʹ͓͍ͯ΋֬ೝͰ͖ɼ੩త෺ମͱಈత෺ମͷͲͪΒʹ͓͍ͯ΋ࢹઢ৘ใ͕༗ޮͰ͋Δ͜ͱ͕Θ͔ΔɽಉҰͷ࣮‫͚͓ʹݧ‬ΔΤϐιʔυࣦഊ‫ݪ‬Ҽͷ͏ͪɼಈత෺
ମͱͷিಥ͕‫ݪ‬ҼͷΤϐιʔυࣦഊ཰Λද 2 ʹࣔ͢ɽ͜͜ͰɼCol. Ped. ͸าߦऀͱͷিಥʹΑΔࣦഊɼCol. Veh.
͸ंͱͷিಥʹΑΔࣦഊͰ͋Δɽද 2 ΑΓɼఏҊख๏͸͢΂ͯͷ৚݅‫ͼٴ‬λεΫʹ͓͍ͯࣗಈंͱͷࣄ‫͕཰ނ‬௿͍͜ͱ͕֬ೝͰ͖Δɽ੒ޭ཰͕௿Լ͍ͯͨ͠ະ஌ఱީΛ‫ؚ‬Ή৚

New town
Dense

New town &weatherฏ‫ۉ‬RegularDenseࣦഊ‫ݪ‬Ҽ
Col. Ped.
Col. Veh.
Col. Ped.
Col. Veh.
Col. Ped.
Col. Veh.
Col. Ped.
Col. Veh.
Col. Ped.
Col. Veh.
Col. Ped.
Col. Veh.
Col. Ped.
Col. Veh.
Col. Ped.
Col. Veh.
Col. Ped.
Col. Veh.

CILRS
9
21
21
57
6
22
14
68
9
30
10
68
6
24
14
70
11.1
45.0

Ours
4
11
16
47
14
18
16
48
9
16
16
58
10
16
20
52
13.1
33.3

݅ʹ͓͍ͯ΋ࣄ‫͕཰ނ‬վળ͞Ε͓ͯΓɼCILRS ʹൺ΂ͯΑΓࣗಈंʹ൓ԠͰ͖͍ͯΔ͜ͱ͕Θ͔ΔɽҰํɼาߦऀ
ͱͷࣄ‫཰ނ‬͸ɼֶश‫͍͓ͯʹڥ؀‬͸վળ͞Ε͍ͯΔ͕ɼະ஌ͷ‫͍͓ͯʹڥ؀‬͸ࣄ‫͕཰ނ‬վળ͍ͯ͠ͳ͍͜ͱ͕Θ͔Δɽ
͜Ε͸ɼֶशσʔληοτ಺ͷาߦऀ͕‫·ؚ‬ΕΔγʔϯ͕ෆ଍͠ɼͦͷଞͷγʔϯͱൺ΂ͯาߦऀʹର͢Δࢹઢਪఆ͕ࠔ೉Ͱ͋Δ͜ͱ͕ߟ͑ΒΕΔɽ

5. ͓ΘΓʹ
ຊ‫Ͱڀݚ‬͸ɼࢹઢ৘ใΛ‫ͨ͠༻׆‬Ұ؏ֶशϕʔεͷࣗಈӡసख๏ΛఏҊͨ͠ɽࢹઢਪఆϞσϧΛ༻͍ͯɼӡసγʔϯʹ͓͚ΔॏཁͳྖҬΛ‫ڧ‬ௐ͢Δ͜ͱͰɼγϛϡϨʔλ্Ͱͷ૸ߦ࣮‫͍͓ͯʹݧ‬ӡసਫ਼౓͕վળ͞ΕΔ͜ͱΛ֬ೝͨ͠ɽࠓ‫ޙ‬͸ɼาߦऀγʔϯͷਫ਼౓޲্΍ࢹઢਪఆϞσϧͷಛ௃ྔΛӡస੍‫ޚ‬Ϟσϧʹར༻ͨ͠ख๏ͷߟҊͳͲΛ‫ݕ‬౼͢Δɽࢀߟจ‫ݙ‬[1] M. Bojarski, et al., “End to End Learning for SelfDriving Cars ”, arXiv preprint arXiv:1704.07911
2016.
[2] F. Codevilla, et al., “Exploring the Limitations of
Behavior Cloning for Autonomous Driving”, ICCV,2019.
[3] A. Dosoviskiy, et al., “CARLA: An Open Urban
Driving Simulator”, CoRL, 2017.

‫੷ۀڀݚ‬[1] ৿‫ܒ‬հ ౳, “ࢹઢ৘ใΛར༻ͨ͠Ұ؏ֶशϕʔεʹΑΔࣗಈӡస੍‫ޚ‬ͷߴਫ਼౓Խ”, ViEW, 2020.
(ଞ 2 ݅)