The 16th Annual Conference of Japanese Society for Artificial Intelligence, 2002

2A1-4

ÁèĝÀ,<´ĄĝÀăėƐù¿-jTĄİReinforcement Learning and Computational Neuroscience- Possible Functions of Neuromodulators in Metalearning ŲŒÇô*1,*2
Kenji Doya
*1

*2

ATR Ġ¤ĔƔÅ°ċ

ATR Human Information Science Laboratories«Ĉě¶ï¸ŗ CREST

CREST, Japan Science and Technology Corporation

The framework of reinforcement learning captures an essential function of the nervous system: to realize behaviors foracquisition of reward. Thus the architectures and algorithms of reinforcement learning can provide important clues as to theorganization and functions of the nervous system. Here I report three such examples: 1) a model of the basal ganglia as thecircuit for reinforcement learning; 2) understanding of the specialization and collaboration of the cerebellum, the basal
ganglia, and the cerebral cortex; 3) working hypotheses about the roles neuromodulators in regulating the metaparameters ofreinforcement learning. The concept of reinforcement learning can provide a common ground for interdisciplinary studies.

sensory
input

EM+*-Ɣą?ų<"6-ÛŰ?ŕæŤ,Ą<)´Ą-ƟŏĮš.ŰƐ8Ġ¤-ÛŰĄã7¦ƙŤ+łƝ?Ń'< 7 7´<. “reinforcement” )Ìƫ.ĚƯ8ĪƐ-ĥ:ý;'ƭ"7-( <´Ą-ÖŤ+Å°-ś(ſ="ApLoQi8AGXHVl.Ġ¤8Ű
Ɛ-ÛŰĄ-ŻŵjF]Qi?Ư<đ(Ƥƪ+ÿ;?Ƨ'=<

1. 





Cerebral cortex
state stStriatum
striosomestate value
V(s)ƸƮıŲŒÇô, ATR Ġ¤ĔƔÅ°ċ, 619-0288 ±ŨƊĽ»īŜÒō 2-2-2, Fax: 0774-95-1259, E-mail:doya@atr.co.jp http://www.atr.co.jp/his/~doyamatrixaction value
Q(s,a)

TD error δtSNc
dopamine neuronsŎŻƂù)Ż-¤,xŚ<ŎŻ¦Ţ. -ƈƒ,9;ª<bGsSsƈ8`sVsZsƍŮƈ+*ďĖ:ŰĦÏ,:-¾><)ř:='" -ĩēò-¨ź.ŞĝÀ-Ŷ( $"-Ɯ-ìÔ?Ƨ'="-śŻ:ŎŻ¦Ţ8ĵůƫ,ĳ{?Ę/[bhs)Ɛù?ƕć<]mrs-Ű©ƺ( <[1]LED Ŧū":qa?)ÛŰ?Mp,Ą"ò -Ċ§,.[bhs]mrs.Ɣą( <OmP,Ŋ'Ŭ<ÛŰĄƲ"Î,.[bhs]mrs. LED -Ŧū,Ŋ'Ŭ<9,+;
OmP,Ŋ'.Ŭ++<=.´ĄƯƻ(Ɣą-Ʀń:-ŀÊ?ƇTD Ñá
δt = rt + V(st) - γ V(st-1)-ě<ƍ)Ƅē,ƴð'<TD Ñá.ƔąƦń)ÛŰĄ-Ƴü-ĄęÝ( ;-ſÆ?½¨,ŎŻ¦Ţ)[bhs]mrs-¨ź?´Ą-ƼĹ5(İƜ<kYpţv='<ġ [1,2][bhs¿-ƌƣƐyŇ+*ÛŰ-´,%+<)ïú.wĵ:ř:='"ŎŻ¦Ţ-N\dP-ķħ[bhs,9$'ĦÏ=<)úÈŤmotor
outputSNr, GPaction at

reward rtThalamusġ ŎŻ¦Ţ-´ĄkYpŎŻƂù-ĖŋƇË st ?7),ĳĕŉ(striatum)(ĖŋŘ¥ģ V(s))ÛŰŘ¥ģ Q(s,a)Áè=ßùƞƩƎ(SNr))Ŗľ¯(GP)?À'íĎ(Thalamus)8Ż-ŰśĤ1-ƹ(ÛŰ at ĴŐ=<ßùƽƛƎ(SNc). TD Ñáδt ?ĳĕŉ,cB[aWH[bhsyŇħ-N\dPķħ,9;Ř¥ģĄ=<,76:=Ɣą-Ʀń) =,¦&ÛŰĴŐŎŻ¦Ţ-¦ƙŤ+¨ź)'Ư=<9,+$"

2. !#PET 8 fMRI +*,9<Ġ¤-ŻŰÁń,9;wĵ.þ,Ű,¥><)Ú:='ƭ"čŻ8ŎŻ¦Ţ
ÌÐĉƯ8uèCjOļå8ƇĔŸř+*Űw-¨ź,7¥><)ƜŽ,+$"[3]4"ĝÀƸƮ?ŝ2<«Ĉ-ğƓ,9;čŻ7ŎŻ¦Ţ7ÜóŸř¨ź-â( <)=<ŎŻƂù-ĵůĵơ,ćƵ?Ŀ<)Ɯ:,=" (ŰĦÏ#(.+čŻ)ŎŻ¦Ţ-Ƣ,¥'Ĝ"+ëÚ-ƼĹ5ƆƬ)='<ŎŻ¦Ţ?´Ą?úË<"6-1-

The 16th Annual Conference of Japanese Society for Artificial Intelligence, 2002-ƹ<kYp.:,ŻĶŉ-¨źƑ-ÉƯ,¥'7ĆƬ+õà?Ƨ'=<
=4(-Å°(čŻ-Ą.ÑáęÝ?7),"µê ;Ą -ƼĹ5(İƜ='<4"ŎŻƂù-]mrs-ŬŴħ.ŷƵ-ŭÁŤħù?7),"µê+Ą -ƼĹ5(İƜ='<=:)Þ><)čŻŎŻ¦ŢŎŻƂù. =!=µê ;Ą´Ąµê+Ą)z+<Ā
ƶ-Ą-ApLoQi,'ĲƠ"ĝÀƹ(<)ƼĹ5Ƌ0đ<ġ [4]
Unsupervised learningġ ŌƇŤ+ĝÀăėƐù¿ßùƽƛƎ(SNc))Ə
łƃơ(VTA)-[bhs¿żłtĳ(DR)-RrZ]s¿Ĭƀ(LC)-_pA[q\os¿ś(S))gC^pZ(M)-ARVpKos¿

outputinput
Cerebral Cortex

Reinforcement learning
rewardBasal Thalamus
Ganglia
substantia
nigra

input
Cerebellum

inferiorolive

outputtargetSupervised learning+

error
input

outputġ čŻ(cerebellum)ŎŻ¦Ţ(basal ganglia)ŎŻƂù(cerebral cortex). =!=µê ;Ą(supervisedlearnning)´Ą(reinforcement learning)µê+Ą(unsupervised learning),Ŵ"ƹÙŁ)N\dP
ķħ-jF]Qi?ñ%
Ʒ/ƎƑ¢ń-´Ą(.Ř¥ģ,9<¦ƙŤ+´Ą-Ƭĸ,=Ėŋ-ĢšŘ(belief state)?
ØĜ<Ɩ÷ƪ:=<-9+ĉƯ.čŻ,µê ;Ą(ų="¡³-ƦńkYp)ŎŻ¦Ţ,´Ą(ų="Ř¥ģ?ŎŻƂù,µê+Ą(ų="=ĖŋƇË?'%+Þ><)(úËź+.( <-9+Ż-Ŏ¹Ť+¨źƑ )¨źŭ Þ,¥ <İ .fMRI + *(ų: =<Ż-*Ű'< )YT? (*ĉƯÛ>='< )Ư,%+<đ(Ƥƪ+ÿ;)+;ų<

3. "´Ą?rfWZĦÏ+*,ťƪ'5'[5,6]ş <).rfWZĄ<wđ,<"6,úÈüöĞĄ-Ɩ?Ą:='<))(<´ĄApLoQi(.Čƭ-ƔąƦń-~Ʊ(γ)ŕæ-nsUi?Â6<¬ ũ(β)Ą-Ņũ
¼ģ(α)+*-jTbnjT -ťĭ+Įš.Ąŏ8¡³ĕÃ,yŇ<"6ň-ĒÞúÈü-îÛçÑ,9<Vm]sI?ƆƬ)<=Å°øqep(.ĨÓ?Ă6'<ĄrfWZő++ĥ-ś,ć'Û+Ŏ+Ưƥ( <}Ɩ>=>=-Ż. -Ą-jTbnjT?œ,Vm]sI'7:>+'7ƚř-¡³-7)(Ʃ+ÛŰ?öưŤ,Ą<)ź( <%4;Ż,. -Ą-jTbnjT?öÍŝį<jTĄ -¨Ùƅ>$'<)Ú:=<
Ż,<jTĄ-Ŕÿ)'Ż:ŎŻƂù
8¦ŢčŻ,×ŪûñņŤ+åƪ?­3ĝÀăėƐù¿ Ú:=<ġ ºŹƑéĪƐŤ«Ĉ-ſő,9;Ʃ+ĝÀăėƐù) -āƨŉŻŵ(-ƑƉ8äƗqep(-åƪ4" -ĺƣ8|
ŧé_WHADZ,9<ÛŰ1-·,¥<YT.Ƙ
Ŏ,ų:='< =:-úÈŤřÆ)ƯƻkYp?eP,
1) [bhs.ƔąƦń:-ŀÊTD Ñáδ2) RrZ]s.ƔąƦń-ò¤PJp~Ʊγ3) _pA[q\os.ÛŰ-nsUi¬ ũβ4) ARVpKos.©-ØĜ-ĄŅũα? =!=ƇËĦÏ'<)İÚ:=<[7]
-İ-ÄĐ,Õ'CRESTŻ?Ļ< -Å°ŏ)'ĄƯƻÅ°nWZ8Mp(-ĪƯúÈĠ¤-ŻŰÁńrfWZúÈ?Ĺ5Þ>"²űÅ°ğ@(<4. ´Ą-ƟŏĮš.9;ƴħź?®6<Ö,7
4"ĪƐ-ť8Ġ¤-ÛŰ-ÉƯ,ž<Å°,7²ŠƼĹ5( ;z+<ƟƑơ?£Ĝ"+Å°-¦Ɓ?
Ƨ<7-)+;ų<$
[1] Schultz, W., Dayan, P., and Montague, P.R.: A neuralsubstrate of prediction and reward. Science, 275, 1593-1599(1997).
[2] Houk, J.C., Adams, J.L., and Barto, A.G.: A model of how
the basal ganglia generate and use neural signals that predict
reinforcement. In J.C. Houk, et al. Eds: Models ofInformation Processing in the Basal Ganglia, pp. 249-270.
MIT Press (1995).
[3] Doya, K.: Complementary roles of basal ganglia andcerebellum in learning and motor control. Current Opinion inNeurobiology, 10, 732-739 (2000).
[4] Doya, K.: What are the computations of the cerebellum, thebasal ganglia, and the cerebral cortex. Neural Networks, 12,961-974 (1999).
[5] Doya, K.: Reinforcement learning in continuous time andspace. Neural Computation, 12, 219-245 (2000).
[6] Doya, K., Kimura, H., and Kawato, M.: Neural mechanismsof learning and control. IEEE Control Systems Magazine,21(4), 42-54 (2001).
[7] Doya, K.: Metalearning and neuromodulation: NeuralNetworks, 15(4), (2002)-2-