Attention ‫ʹߏػ‬ΑΔࢹ֮తઆ໌ΛՄೳͱͨ͠ਂ૚‫ڧ‬Խֶशʹؔ͢Δ‫ڀݚ‬ࢦಋ‫ڭ‬तɿࢁԼོٛ
TP19001 ൘୩ӳయ
1. ͸͡ΊʹΈࠐΈ૚ͱ Sigmoid ؔ਺Λద༻͢Δ͜ͱͰ֫ಘ͢Δɽਂ૚‫ڧ‬Խֶश͸ɼΤʔδΣϯτ͕‫ͱڥ؀‬ͷΠϯλϥΫγϣϯΛ௨ͯ͡ಘΔใुΛཔΓʹɼঢ়‫ʹگ‬Ԡͯ͡࠷దͳߦಈબ୒͢ΔͨΊͷํࡦΛֶश͢Δख๏Ͱ͋Δɽํࡦ͸ωοτϫʔΫͰද‫͞ݱ‬Ε͓ͯΓɼωοτϫʔΫ಺෦ͷԋࢉ͸ෳࡶͰ͋ΔɽͦͷͨΊɼΤʔδΣϯτͷߦಈબ୒ʹରͯ͠൑அࠜ‫ڌ‬Λղੳ͢Δ͜ͱ͸ඇৗʹࠔ೉Ͱ͋Δɽຊ‫Ͱڀݚ‬͸ɼਂ૚‫ڧ‬Խֶशͷ୅දతͳख๏Ͱ͋Δ Asynchronous Advantage Actor-Critic (A3C) [1] ʹ Attention‫ߏػ‬Λಋೖͨ͠ Mask-Attention A3C (Mask A3C) ΛఏҊ͢ΔɽMask A3C Ͱ͸ɼํࡦͱঢ়ଶՁ஋ʹର͢ΔΤʔδΣϯτͷ஫ࢹྖҬΛද‫ ͨ͠ݱ‬Mask-attention Λ֫ಘͰ͖Δɽਪ࿦࣌ʹ Mask-attention ΛՄࢹԽ͢Δ͜ͱͰɼΤʔδΣϯτͷߦಈબ୒ʹର͢Δࢹ֮తઆ໌ͷ࣮‫ݱ‬Λ໨తͱ͢Δɽ
·ͨɼAttention ‫ߏػ‬ͷಋೖʹΑΓɼMask-attention Λߟֶྀͯ͠श͢Δ͜ͱͰɼΤʔδΣϯτͷੑೳ޲্ΛਤΔɽ

3.2 Attention ‫ߏػ‬

2.Asynchronous Advantage Actor-Critic

Mask A3C ͷ༗ޮੑΛ֬ೝ͢ΔͨΊɼOpenAI Gym [3]ͷήʔϜλεΫΛ༻͍ͯධՁ࣮‫ݧ‬Λߦ͏ɽ

A3C [1] ͸ɼAsynchronous ͱ Advantage Λಋೖͨ͠
Actor-Critic ๏ϕʔεͷਂ૚‫ڧ‬Խֶशख๏Ͱ͋ΔɽAsynchronous ͸ෳ਺‫͚͓ʹڥ؀‬Δඇಉ‫Ͱظ‬ͷύϥϝʔλߋ৽ɼ
Advantage ͸਺εςοϓઌͷใुΛߟֶྀͨ͠शͰ͋Δɽ
A3C ͷωοτϫʔΫߏ଄͸ɼ৞ΈࠐΈ૚ʹΑΓಛ௃ϚοϓΛநग़͢Δ Feature extractorɼํࡦΛग़ྗ͢Δ Policybranchɼঢ়ଶՁ஋Λग़ྗ͢Δ Value branch ͔Βߏ੒͞ΕΔɽํࡦ͸͋Δঢ়ଶʹ͓͍ͯબ୒͢Δߦಈͷ֬཰෼෍Ͱ͋Δɽঢ়ଶՁ஋͸͋Δঢ়ଶʹ͓͚Δใुͷ‫ظ‬଴஋Ͱ͋Γɼ͋Δঢ়ଶʹ͍Δ͜ͱͷՁ஋Λද͢ɽ

3.Mask-Attention A3CΤʔδΣϯτͷߦಈબ୒ʹର͢Δ൑அࠜ‫ڌ‬Λղੳ͢Δ
ͨΊɼA3C ʹ Attention ‫ߏػ‬Λಋೖͨ͠ Mask-Attention
A3C (Mask A3C) ΛఏҊ͢ΔɽMask A3C Ͱ͸ɼPolicybranch ͱ Value branch ʹର͠ Attention ‫ߏػ‬Λಋೖ͢
Δ͜ͱͰɼ֤ϒϥϯνͷग़ྗʹରͯ͠஫ࢹͨ͠ྖҬΛද͢
Mask-attention Λ֫ಘ͢Δɽਪ࿦࣌ʹ͓͚Δ֤ϒϥϯνͷ Mask-attention ΛՄࢹԽ͢Δ͜ͱͰɼํࡦͱঢ়ଶՁ஋ͷҟͳΔ 2 ͭͷࢹ఺͔ΒɼΤʔδΣϯτͷߦಈબ୒ʹର͢
Δࢹ֮తઆ໌Λ࣮‫͢ݱ‬Δɽ

3.1 ωοτϫʔΫߏ଄

Conv.

Sigmoidਤ 1 ʹఏҊ͢Δ Mask A3C ͷωοτϫʔΫߏ଄Λࣔ͢ɽ
Mask A3C ͸ɼFeature extractorɼPolicy branchɼValuebranchɼAttention ‫͔ߏػ‬Βߏ੒͞ΕΔɽैདྷͷ A3C Ͱ͸ɼ࣌‫ྻܥ‬৘ใΛߟྀ͢ΔͨΊɼFeature extractor ʹ LSTMΛ༻͍Δɽ͔͠͠ɼMask A3C ʹ LSTM Λಋೖ͢Δͱɼೖྗը૾ʹର͢Δۭؒ৘ใ͕ܽམ͢ΔͨΊɼMask-attentionΛ֫ಘͰ͖ͳ͍ɽͦ͜Ͱɼۭ࣌ؒ৘ใΛߟྀͰ͖Δ Convolutional LSTM (ConvLSTM) [2] Λಋೖ͢Δɽϒϥϯν i ͷ Mask-attention Mi ͸ɼFeature extractor ͔Βग़ྗ͞ΕΔಛ௃Ϛοϓ Fi ʹର͠ɼ1 × 1× # of chanels ͷ৞#$FCState value

FCConv.

&$'&$

PolicyPolicy branch
Conv.

ReLU

Conv.

Attentionmechanism

Sigmoid

Input

Value branch

ConvLSTM

Conv.

Maxpooling + ReLU

Conv.

Attentionmechanism
Maxpooling + ReLU

Conv.

Maxpooling + ReLU!"

ReLU

Mask-attention&%'&%Feature extractor#%
Mask-attention

ਤ 1: Mask A3C ͷωοτϫʔΫߏ଄

Mask A3C Ͱ͸ɼPolicy branch ͱ Value branch ʹAttention ‫ߏػ‬Λಋೖ͢Δɽ͜ΕʹΑΓɼ֫ಘͨ͠ Maskattention Mi Λߟྀ͠ɼํࡦ‫ͼٴ‬ঢ়ଶՁ஋Λग़ྗ͢ΔɽAttention ‫ߏػ‬͸ɼϒϥϯν i ಺ʹ͓͚Δதؒ૚ͷಛ௃ϚοϓFi ʹର͠ɼMask-attention Mi Λ༻͍ͯϚεΫॲཧΛߦ͏ɽಛ௃Ϛοϓʹର͢Δ Mask-attention Λ༻͍ͨϚεΫॲཧΛࣜ (1) ʹࣔ͢ɽ͜͜Ͱɼst ͸ೖྗͰ͋Δ‫ݱ‬ঢ়ଶɼFi (st )͸ϒϥϯν i ಺ʹ͓͚Δதؒ૚ͷಛ௃ϚοϓɼMi (st ) ͸ϒϥϯν i ʹ͓͚Δ Mask-attentionɼFi′ (st ) ͸ϚεΫॲཧ‫ޙ‬ͷಛ௃ϚοϓͰ͋Δɽ
Fi′ (st ) = Fi (st )ɾMi (st )(1)

4. ධՁ࣮‫ݧ‬

4.1 ࣮‫֓ݧ‬ཁ࢖༻͢ΔήʔϜ͸ɼBreakout ͱ Ms.Pac-ManɼSpaceInvaders ͷ 3 छྨͰ͋Δɽൺֱख๏͸ɼA3C ͱ Mask A3Cɼ֤ϒϥϯνͷΈʹରͯ͠ Attention ‫ߏػ‬Λಋೖͨ͠ MaskA3C (Policy Mask A3CɼValue Mask A3C) ͷ‫ ܭ‬4 ͭͰ͋Δɽೖྗ͸ήʔϜը໘ͷάϨʔεέʔϧը૾ͱ͠ɼग़ྗ
Ͱ͋Δߦಈ͸֤ήʔϜʹ͓͚Δૢ࡞ͱ͢Δɽֶश৚݅͸
worker ਺Λ 35ɼֶश܎਺Λ 0.0001ɼׂҾ཰Λ 0.99 ͱ͢Δɽֶशऴྃ৚݅͸ global step ਺͕ 1.0 × 108 ʹ౸ୡͨ͠
৔߹ͱ͢Δɽ·ͨɼΤϐιʔυͷऴྃ৚݅͸֤ήʔϜʹ͓
͚Δ 1 ϓϨΠऴྃɼ‫ ͼٴ‬step ਺͕ 1.0 × 104 ʹ౸ୡͨ͠৔
߹ͱ͢Δɽ

4.2 είΞൺֱ֤ήʔϜλεΫʹ͓͚Δ 100 Τϐιʔυؒͷ࠷େ/ฏ‫ۉ‬είΞΛද 1 ʹࣔ͢ɽද 1 ͔ΒɼBreakout ʹ͓͚Δ࠷େείΞ͸શख๏ʹ͓͍ͯ 864 Ͱ͋Δɽ͜ͷείΞ͸ɼBreakoutͰ֫ಘͰ͖Δ࠷ߴείΞͰ͋Δɽ·ͨɼBreakout ʹ͓͚Δฏ‫ۉ‬είΞ͸ɼMask A3C ͕ A3C ͱൺֱ͠΄΅ಉ౳Ͱ͋ΔɽMs.Pac-Man Ͱ͸ɼ࠷େ/ฏ‫ۉ‬είΞ‫ ʹڞ‬Mask A3C͕࠷΋ߴ͍είΞͰ͋ΔɽSpace Invaders Ͱ͸ɼ࠷େείΞ͸ Policy Mask A3Cɼฏ‫ۉ‬είΞ͸ Mask A3C ͕࠷΋ߴ͍είΞͰ͋ΔɽBreakout ͸ύυϧͰϘʔϧΛଧͪฦ͢ͷΈͰ͋Γɼ֎తཁҼͷͳ͍୯७ͳλεΫͰ͋ΔɽͦͷͨΊɼA3C ͱ Mask A3C ͕ಉ౳ͷείΞͰ͋ͬͨͱߟ͑ΒΕΔɽҰํɼMs.Pac-Man ͱ Space Invaders ͸ɼఢͳͲͷ֎తཁҼΛߟྀͯ͠ߦಈΛબ୒͢Δඞཁ͕͋ΔɽPolicy
branch ʹ Attention ‫ߏػ‬Λಋೖͨ͠ Policy Mask A3C ͱ
Mask A3C Ͱ͸ɼΫοΩʔ΍ΠϯϕʔμʔͳͲɼํࡦʹؔ࿈ͨ͠ྖҬΛ‫ڧ‬ௐ͢ΔɽͦͷͨΊɼA3C ͱൺֱ͠είΞ͕޲্ͨ͠ͱߟ͑ΒΕΔɽ

4.3 Mask-attention Λ༻͍ͨࢹ֮తઆ໌֤ήʔϜʹ͓͚Δ Mask-attention ͷՄࢹԽྫΛਤ 2 ʹࣔ͢ɽਤ 2(a) ͷ Policy ͔ΒɼFrame 1 Ͱ͸Ϙʔϧͷਐߦํ޲Λ஫ࢹ͍ͯ͠ΔɽFrame 2 Ͱ͸ύυϧΛ‫ؚ‬ΊͨϘʔϧपғΛ஫ࢹ͠ɼϘʔϧΛଧͪฦͨ͠‫͋Ͱޙ‬Δ Frame 3 Ͱ͸஫ࢹ͍ͯ͠ΔྖҬ͕ͳ͍ɽ͔͜͜ΒɼBreakout Ͱ͸ϘʔϧΛଧͪฦ͢௚લʹ͓͍ͯɼϘʔϧͷਐߦํ޲Λ༧ଌ͠ύυϧΛ੍‫͍ͯ͠ޚ‬Δͱߟ͑ΒΕΔɽਤ 2(b) ͷ Policy ͔ΒɼFrame 1 Ͱ͸ύοΫϚϯपғΛ஫ࢹ͍ͯ͠ΔɽFrame2 Ͱ͸ը໘಺ʹଘࡏ͢ΔΫοΩʔΛ஫ࢹ͠ɼFrame 3 Ͱ͸ Frame 2 ʹ͓͚Δ஫ࢹྖҬʹύοΫϚϯ͕Ҡಈ͍ͯ͠Δɽ͔͜͜ΒɼMs.Pac-Man Ͱ͸ύοΫϚϯͷपғΛ‫ܯ‬ռ͠ͳ͕ΒɼΫοΩʔ΁޲͔͏Α͏ʹ੍‫͍ͯ͠ޚ‬Δͱߟ͑ΒΕΔɽਤ 2(c) ͷ Policy ͔ΒɼFrame 1 ͰΠϯϕʔμʔΛ஫ࢹ͠ɼߦಈ͸߈ܸΛબ୒͍ͯ͠Δɽͦͯ͠ɼFrame 3

Point 1

Frame 1

Frame 2

Point 2

Point 1

Frame 3

PolicyPoint 3

Point 2

Point 3

State value

State value graph(a) Breakout: ന໼ҹ͸Ϙʔϧͷਐߦํ޲Λࣔ͢ɽ
Point 1

Frame 1

Frame 2

Point 1

Frame 3

Point 3

Point 3

Point 2

PolicyPoint 2

State value graphState value(b) Ms.Pac-Man: ന໼ҹ͸ύοΫϚϯͷਐߦํ޲Λࣔ͢ɽ
Point1

Frame 1

Frame 2

Point 1

Frame 3

Point 2

PolicyPoint2

ɹ

Point3

Point 3

State value graphState value(c) Space Invaders: ന໼ҹ͸ϏʔϜͷਐߦํ޲Λࣔ͢ɽਤ 2: Mask-attention ͷՄࢹԽྫ: Policy ʹ͓͚ΔԼ෦ͷίϯτϩʔϥ͸ɼ‫ݱ‬ϑϨʔϜͰϞσϧ͕ਪ࿦ͨ͠ߦಈͰ͋Δɽද 1: ֤ήʔϜ͓͚Δ 100 Τϐιʔυؒͷ࠷େ/ฏ‫ۉ‬είΞ:֤ख๏Ͱ 5 ࢼߦֶͣͭश͠ɼฏ‫ۉ‬είΞ͕࠷΋ߴ͍ϞσϧͷείΞΛࣔ͢ɽmax/mean ͸࠷େ/ฏ‫ۉ‬είΞͰ͋Δɽ
Ms.Pac-Man

Space Invaders

Att. mechanismmax

meanmax

meanmaxmean

Policy

864

662.0

5380

4573.3

19505

18531.8

864

595.8

6330

4833.819860

19102.8!

864

606.9

4830

4044.5

19675

18537.8

!

864

640.0

6610

5314.1

19810

19212.5

Att. mechanism
Policy

Value

A3C!
ఏҊख๏

!

ද 2: Mask-attention ͷ൓సʹΑΔείΞൺֱɿnormal ͸ Mask-attention Λ൓స͠ͳ͍৔߹ɼin- ɹverse ͸൓సͨ͠৔߹ɼrandom ͸ϥϯμϜʹߦಈબ୒ͨ͠৔߹Ͱ͋Δɽ

Breakoutʹ͓͍ͯ Frame 1 Ͱ஫ࢹͨ͠ΠϯϕʔμʔΛܸഁ͍ͯ͠
Δɽ͔͜͜ΒɼSpace Invaders Ͱ͸ܸഁ͢ΔΠϯϕʔμʔΛೝࣝ͠ɼΤʔδΣϯτΛ੍‫͍ͯ͠ޚ‬Δͱߟ͑ΒΕΔɽ·ͨɼਤ 2(a)(b)(c) ͷ State value ͔ΒɼBreakout Ͱ͸ϒϩοΫɼMs.Pac-Man Ͱ͸ΫοΩʔɼSpace Invaders Ͱ͸ΠϯϕʔμʔΛ஫ࢹ͍ͯ͠Δɽͦͯ͠ɼ஫ࢹͨ͠෺ମͷ‫ݮ‬গʹ߹Θͤ஫ࢹྖҬ͕ॖখ͍ͯ͠Δɽ͜ΕΒͷ݁Ռ͔Βɼ
Policy ͷ Mask-attention ͸‫ݱ‬ঢ়ଶʹର͠ߦಈʹ௚݁͢Δ
෺ମɼState value ͷ Mask-attention ͸είΞʹ‫د‬༩͢Δ෺ମΛද͍ͯ͠Δͱߟ͑ΒΕΔɽ

4.4 Mask-attention ͷ༗ޮੑ
Mask A3C ʹΑΔߦಈͷࢹ֮తઆ໌Λߦ͏ʹ͋ͨΓɼ
Mask-attention ͕Ϟσϧͷग़ྗͰ͋Δํࡦʹରͯ͠༗ӹͳྖҬΛද͍ͯ͠Δ͔‫͢ূݕ‬Δɽ‫ํূݕ‬๏ͱͯ͠ɼMaskA3C ʹ͓͚Δ Policy branch ͷ Mask-attention Λ൓సͨ͠ϚοϓΛ࡞੒͠ɼͦͷϚοϓΛ Attention ‫͍ͨ༻ʹߏػ‬৔߹ͷείΞΛࢉग़͢ΔɽMask-attention Λ൓స͢Δ৔߹ͱ൓స͠ͳ͍৔߹ʹ͓͚ΔείΞΛൺֱ͢Δ͜ͱͰɼMaskattention ͕ߦಈͷࢹ֮తઆ໌ʹ༗ޮͰ͋Δ͜ͱΛ֬ೝ͢
Δɽද 2 ʹɼMask-attention ͷ൓సʹΑΔείΞൺֱΛࣔ
͢ɽද 2 ͔ΒɼશήʔϜʹ͓͍ͯ inverse ͕ normal ͱൺֱͯ͠ஶ͘͠είΞ͕௿Լ͠ɼrandom ͱಉ౳ͷείΞͰ͋Δɽ͕ͨͬͯ͠ɼPolicy branch ʹ͓͚Δ Mask-attention

normal!
!

Breakout

Value!
random

Ms.Pac-Man

Space Invaders

max

mean

max

mean

max

mean

864

595.8

6630

4833.819860

19102.8

inverse

4

2.2

290

268.9

805

306.9

normal

864

640.0

6610

5314.1

19810

19212.5

inverse

5

1.8

410

194.4

915

420.2

5

1.2

1080

247.8

460

142.1

͸ɼߴείΞΛ֫ಘ͢Δߦಈʹରͯ͠༗ӹͳྖҬΛ֫ಘͨ͠ͱ‫͑ݴ‬Δɽ

5. ͓ΘΓʹ
ຊ‫Ͱڀݚ‬͸ɼਂ૚‫ڧ‬Խֶशͷ୅දతͳख๏Ͱ͋Δ A3C ʹAttention ‫ߏػ‬Λಋೖͨ͠ Mask A3C ΛఏҊͨ͠ɽMaskA3C Ͱ͸ɼਪ࿦࣌ʹ Mask-attention ΛՄࢹԽ͢Δ͜ͱͰɼΤʔδΣϯτͷߦಈબ୒ʹର͢Δ൑அࠜ‫ڌ‬ͷࢹ֮తઆ໌Λ࣮‫ͨ͠ݱ‬ɽࠓ‫ޙ‬͸ɼϩϘοτ੍‫Ͳͳޚ‬ͷߋʹෳࡶͳλεΫ΁ͷద༻ͱՄࢹԽʹऔΓ૊Ή༧ఆͰ͋Δɽࢀߟจ‫ݙ‬[1] V. Minh, et al., “Asynchronous Methods for Deep Reinforcement Learning”, ICML, 2016.
[2] X. Shi, et al., “Convolutional LSTM Network: A Machine Learning Approach for Precipitation Nowcasting”,
NeurIPS, 2015.
[3] G. Brockman, et al., “OpenAI Gym”, arXiv preprint
arXiv:1606.01540, 2016.

‫੷ۀڀݚ‬[1] ൘୩ӳయ ౳, “A3C ʹ͓͚Δ Attention ‫ߏػ‬Λ༻͍ͨࢹ֮తઆ໌”, ਓ޻஌ೳֶձશࠃେձ, 2020. (ֶੜ঑ྭ৆ड৆)(ଞ 2 ݅)