Kenji Doya's Publications

Metalearning and Neuromodulators
Asahi S., Okamoto, Y., Okada, G., Morinobu, S., Yamawaki, S., , Doya K. (2002).
Relationship between brain activation during GO/NOGO task andimpulsiveness: A fMRI study. 32nd Annual Meeting, Society for Neuroscience,
Orlando, USA.
Doya K. (2003). A computational theory of neuromodulation. International
Symposium "New Horizons in Molecular Sciences and Systems: An
Integrated Approach," 50.
Doya K. (2003). Enjoy now or strive for future: Neural mechanisms of reward
prediction at different time scales. Summer Program 2003 Progarm and
Abstracts, 36.
Doya K. (2002). Metalearning and neuromodulation. Neural Networks, 15,495-506.
Doya K. (2002). Computational Models of Neuromodulation. Neural Networks
2002 Special Issue on Computational Models of Neuromodulation.
Doya K. (2001). Metalearning and neuromodulation. CREST Workshop on
Metalearning and Neuromodulation, Seika, Kyoto, 6.
Doya K. (2000). Metalearning, neuromodulation, and emotion. Hatano G,
Okada N, Tanabe H, Affective Minds, Elsevier Science, 101-104.
Doya K. (2000). A possible role of serotonin in regulating the time scale ofreward prediction. Serotonin Conference.
Doya K. (2000). Possible roles of neuromodulators in the regulation oflearning processes. 30th Annual Meeting, Society for Neuroscience.
Doya K. (1999). Metalearning, neuromodulation and emotion. 13th Toyota
Conference on Affective Minds, Mikkabi, Japan, 46-47.

Doya K., Okada G., Ueda K., Okamoto Y., Yamawaki S. (2001). Prediction ofshort- and long-term reward: A functional MRI study with a Markov decision
problem. 31st Annual Meeting, Society for Neuroscience, San Diego, USA.
Ito, M., Doya, K., Shirao, T., Sekino, Y. (2004). Fos imaging reveals that the
supramammillary nucleus enhances hippocampal activity of rats placed in anovel open field. Society for Neuroscience 34th Annual Meeting, 96.
Okada G., Okamoto Y., Ueda K., Morinobu S., Yamawaki S., Doya K. (2002).
Selection between small, immediate rewards and large, delayed rewards in
prediction of future reward: A functional magnetic resonance imaging study.
8th International Conference on Functional Mapping of the Human Brain,
Sendai, Miyagi.
Okada G., Okamoto Y., Ueda K., Yamashita H., Kagaya A., Morinobu S.,
Yamawaki S., Doya K. (2001). Localization of brain activity in prediction of
future reward using fMRI and MEG. 31st Annual Meeting, Society for
Neuroscience, San Diego, USA.
Schweighofer, N., Doya, K., Kuroda, S. (2004). Cerebellar aminergic
neuromodulation: towards a functional understanding. Brain research reviews,
44, 103-116.
Schweighofer,N., Tanaka, S., Asahi, S., Okamoto, Y., Doya, K., Yamawaki, S.
(2004). An fMRI study of the delay discounting of reward after tryptophan
depletion and loading. 1: decision-making. Society for Neuroscience 34th
Annual Meeting, 97.
Schweighofer N., Doya K. (2003). Meta-learning of reinforcement learning.
Neural Networks, 16, 5-9.
Schweighofer N., Doya K. (2002). A biologically plausible computational modelof meta-learning in reinforcement learning. 32nd Annual Meeting, Society for
Neuroscience, Orlando, USA.
Tanaka, S., Doya, K., Okada, G., Ueda, K., Okamoto, Y., Yamawaki, S. (2004).
Prediction of immediate and future rewards differentially recruits

cortico-basal ganglia loops. Nature Neuroscience, 7(8), 887-893.
Tanaka, S., Schweighofer,N., Asahi, S., Okamoto, Y., Yamawaki, S., Doya, K.
(2004). An fMRI study of the delay discounting of reward after tryptophan
depletion and loading. 2: reward expectation. Society for Neuroscience 34th
Annual Meeting, 98.
Tanaka S., Doya K., Okada G., Ueda K., Okamoto Y., Yamawaki S. (2004).
Different cortico-basal ganglia loops specialize in reward prediction ondifferent time scales. Advances in Neural Information Processing Systems16,
701-708, MIT Press
Tanaka S., Doya K., Okada G., Ueda K., Okamoto Y., Yamawaki S. (2003).
Prediction of immediate and future rewards differentially recruits
cortico-basal ganglia loops. Society for Neuroscience 33rd Annual Meeting,
58.
Ueda Y., Samejima K., Doya K., Kimura M. (2002). Reward value-dependentstriate neuron activity of monkey performing trial-and-error behavioral
decision task. 32nd Annual Meeting, Society for Neuroscience, Orlando, USA.
Specialization and Integration of Brain Areas
Doya K. (2001). Robotic neuroscience: A synthetic approach to the brain.
Neuroscience Research, Supplement, 24, S16.
Doya K., Kimura H., Kawato M. (2001). Neural mechanisms of learning andcontrol. IEEE Control Systems Magazine, 21, 42-54.
Doya K., Kimura H., Miyamura A. (2001). Motor control: Neural models andsystem theory. International Journal of Applied Mathematics and Computer
Science, 11, 101-128.
Kimura H., Doya K. (2000). Motor control: Neural models and system theory.
14th International Symposium on Mathematical Theory and Networks andSystems, 232.
Reinforcement Learning

Doya K. (2000). Reinforcement learning in continuous time and space. NeuralComputation, 12, 219-245.
Doya K. (1997). Efficient nonlinear control with actor-tutor architecture.
Mozer MC, Jordan MI, Petsche T, Advances in Neural Information Processing
Systems 9, MIT Press, 1012-1018.
Doya K. (1996). Temporal difference learning in continuous time and space.
Touretzky DS, Mozer MC, Hasselmo ME, Advances in Neural Information
Processing Systems 8, MIT Press, 1073-1079.
Doya K. (1996). Reinforcement learning in animals and robots. International
Workshop on Brainware, 69-71.
Koike Y., Doya K. (1999). Multi state estimation reinforcement learning for
driving model. IEEE International Conference on System, Man andCybernetics, Tokyo, V, 504-509.
Morimoto, J., Doya, K. (2005). Robust reinforcement learning. NeuralComputation, 17, 335-359.
Morimoto J., Doya K. (2001). Robust reinforcement learning. Advances in
Neural Information Processing Systems 13, MIT Press, 1061-1067.
Morimoto J., Doya K. (2000). Robust reinforcement learning. Technical
Report of IEICE, NC2000-49, 59-66.
Nakahara H., Doya K. (1998). Near saddle-node bifurcation behavior as
dynamics in working memory for goal-directed behavior. Neural Computation,
10, 113-132.
Bayesian Inference
Samejima K., Doya K., Ueda K., Kimura M. (2004). Estimating internal variables
and parameters of a learning agent by a particle filter. Advances in Neural
Information Processing Systems16, 1335-1342, MIT Press.
Sato, M., Yoshioka, T., Kajiwara, S., Toyama, K., Goda, N., Doya, K., Kawato, M.

(2004). Hierarchical bayesian estimation for MEG inverse problem.
NeuroImage, 23, 806-826.
Cyber Rodent
Capi, G., Doya, K. (2005). Evolution of neural architecture fitting
environmental dynamics. Adaptive Behavior, 13, 53-66.
Capi G., Doya K. (2003). Evolving recurrent neural controllers for sequential
tasks - A parallel implementation. Congress on Evolutionary Computation, 1,
514-519.
Capi G., Uchibe E., Doya K. (2003). Selection of neural architecture and theenvironment complexity. Dynamic Systems Approach for Embodiment andSociality From Ecological Psychology to Robotics, 6, 311-317. Advanced
Knowledge International.
Capi G., Uchibe E., Doya K. (2002). Selection of neural architecture and theenvironment complexity. The 3rd International Symposium on Human and
Artificial Intelligence Systems: Dynamic Systems Approach for Embodimentand Sociality, Fukui, Japan, 231-237.
Doya K., Uchibe E. (2005). The Cyber Rodent project: Exploration of adaptive
mechanisms for self-preservation and self-reproduction. Adaptive Behavior.
Doya K. (2003). Cyber rodents: Self-preserving, self-reproducing robotic
colony. ATR Up to Date, Summer 2003, 12-13.
Elfwing S., Uchibe E., Doya K. (2003). An Evolutionary Approach to Automatic
Construction of the Structure in Hierarchical Reinforcement Learning.
Genetic and Evolutionary Computation - GECCO 2003 Proceedings, Part 1,
Chicago, IL, Springer, GECCO 2003, LNCS 2723, 507-509.
Eriksson A., Capi G., Doya K. (2003). Evolution of meta-parameters in
reinforcement learning algorithm. IEEE/RSJ IROS.
Learning to Stand Up

Morimoto J., Doya K. (2001). Acquisition of stand-up behavior by a real robot
using hierarchical reinforcement learning. Robotics and Autonomous
Systems, 36, 37-51.
Morimoto J., Doya K. (2000). Acquisition of stand-up behavior by a real robot
using hierarchical reinforcement learning. 17th International Conference on
Machine Learning, 1, 623-630.
Morimoto J., Doya K. (1999). Hierarchical reinforcement learning for motion
learning: learning "stand-up" trajectories. Advanced Robotics, 13, 267-268.
Morimoto J., Doya K. (1998). Reinforcement learning of dynamic motor
sequence: Learning to stand up. Proceedings of IEEE/RSJ International
Conference on Intelligent Robots and Systems, 3, 1721-1726.
Morimoto J., Doya K. (1998). Hierarchical reinforcement learning of
low-dimensional subgoals and high-dimensional trajectories. The 5thInternational Conference on Neural Information Processing, 2, 850-853.
Multiple Models
Bapi R.S., Doya K. (2001). Multiple forward model architecture for sequence
processing. Sun R, Giles L, Sequence Learning: Paradigms, Algorithms, andApplications, Springer Verlag, 309-320.
Doya K., Katagiri K., Wolpert D.M., Kawato M. (2000). Recognition andimitation of movement paterns by a multiple predictor-controller
architecture. Technical Report of IEICE, TL2000-11, 33-40.
Doya K., Samejima K., Katagiri K., Kawato M. (2002). Multiple model-based
reinforcement learning. Neural Computation, 14, 1347-1369.
Doya K., Samejima K., Katagiri K., Kawato M. (2001). Task decomposition andimitation by MOSAIC architecture. HFSP Arundel Meeting / Wolpert Group,
Arundel, Canada
Doya K., Samejima K., Katagiri K., Kawato M. (2000). Multiple model-based
reinforcement learning. Japan Science and Technology Corporation.

Samejima K., Doya K., Kawato M. (2003). Inter-module credit assignment inmodular reinforcement learning. Neural Networks, 16, 985- 994.
Samejima K., Doya K., Kawato M. (2002). Inter-module credit assignment inmodular reinforcement learning. Neural Networks.
Uchibe, E., Doya, K. (2004). Competitive-cooperative-concurrent
reinforcement learning with importance sampling. The Eighth International
Conference on the Simulation of Adaptive Behavior, 287-296.
Communication
Doya K., Sugimoto N., Wolpert D.M., Kawato M. (2003). Selecting Optimal
Behaviors Based on Contexts. International Symposium on Emergent
Mechanisms of Communication, Awaji, 19-23.
Nagayuki Y., Ishii S., Doya K. (2000). Multi-agent reinforcement learning: anapproach based on the other agent's internal model. Fourth International
Conference on Multi-Agent Systems, 215-221.
Nagayuki Y., Ishii S., Ito M., Shimohara K., Doya K. (2000). A multi-agent
reinforcement learning method with the estimation of the other agent's
actions. Fifth International Symposium on Artifical Life and Robotics, 1,
255-259.
Wolpert D.M., Doya K., Kawato M. (2003). A unifying computational framework
for motor control and social interaction. Philosophical Transactions of theRoyal Society, 358.
Cerebellum
Kuroda S., Yamamoto K., Miyamoto H., Doya K., Kawato M. (2001). Statisticalcharacteristics of climbing fiber spikes necessary for efficient cerebellar
learning. Biological Cybernetics, 84, 183-192.
Kuroda S., Yamamoto K., Miyamoto H., Doya K., Kawato M. (2000). Statisticalcharacteristics of climbing fiber spikes necessary for efficient cerebellar
learning. Kawato Dynamic Brain Project.

Schweighofer, N., Doya, K., Fukai, H. , Chiron, Jean V., Furukawa, T., Kawato,
M. (2004). Chaos may enhance information transmission in the inferior olive.
Proceedings of the National Academy of Sciences, USA, 101(13), 4655-4660.
Schweighofer N., Doya K., Kawato M. (1999). Electrophysiological properties
of inferior olive neurons: A compartmental model. Journal of Neurophysiology,
82, 804-817.
Schweighofer N., Doya K., Kawato M. (1998). A model of theelectrophysiological properties of the inferior olive neurons. 28th Annual
Meeting, Society for Neuroscience, 24, 667.
Schweighofer N., Doya K., Lay F. (2001). Unsupervised learning of granule cellsparse codes enhances cerebellar adaptive control. Neuroscience, 103,
35-50.
Sequence Learning and Basal Ganglia

Bapi R.S., Doya K. (1999). MFM: Multiple forward model architecture forsequence processing. IJCAI'99 Workshop on Sequence Learning, Stockholm,
Sweden.
Bapi R.S., Doya K. (1998). A sequence learning architecture based on
cortico-basal ganglionic loops and reinforcement learning. The 5thInternational Conference on Neural Information Processing, 1, 260-263.
Bapi R.S., Doya K. (1998). Evidence for effector independent and dependent
components in motor sequence learning. 28th Annual Meeting, Society for
Neuroscience, 24, 167.
Bapi R.S., Doya K., Harner A.M. (2000). Evidence for effector independent anddependent representations and their differential time course of acquisition
during motor sequence learning. Experimental Brain Research, 132, 149-62.
Bapi R.S., Doya K., Harner A.M. (1999). Visual and motor representations forsequence learning. Japan Science and Technology Corporation.

Bissmarck, F., Nakahara, H., Doya, K., Hikosaka, O. (2005). Responding tomodalities with different latencies. Advances in Neural Information
Processing Systems, MIT Press.
Bissmarck F., Nakahara H., Doya K., Hikosaka O. (2003). Parallel network
mechanisms for motor sequence acquisition in real time. IEICE Technical
Report, 102, 113-118.
Doya K. (2001). Specialization of cerebellum, basal ganglia, and cerebral
cortex in supervised, reinforcement, and unsupervised learning. Stockholm
Workshop on Computational Neuroscience, Stockholm, 13.
Doya K. (2000). Complementary roles of basal ganglia and cerebellum inlearning and motor control. Current Opinion in Neurobiology, 10, 732-739.
Doya K. (1999). What are the computations of the cerebellum, the basal
ganglia, and the cerebral cortex. Neural Networks, 12, 961-974.
Doya K. (1999). Multiple representation and algorithms for sequence learning.
2nd International Conference on Cognitive Science, Tokyo, 17-19.
Doya K. (1998). Integration of cortical, cerebellar and basal ganglionicmodules specialized in unsupervised, supervised and reinforcement learning.
International Basal Ganglia Society 6th Triennial Meeting, 27.
Doya K. (1997). How basal ganglia, cerebellum and cerebral motor areas work
together in sequential control tasks. Neural Control of Movement, 7th Annual
Meeting Abstracts, 28.
Doya K. (1996). An integrated model of basal ganglia and cerebellum insequential control tasks. 26th Annual Meeting, Society for Neuroscience, 22,2029.
Haruno, M., Kuroda, T., Doya, K. , Toyama, K. , Kimura, M. , Samejima, K. ,Imamizu, H. , Kawato, M. (2004). A neural correlate of reward-based
behavioral learning in caudate nucleus: a functional magnetic resonanceimaging study of a stochastic decision task. Journal of Neuroscience, 24(7),1660-1665.

Haruno M., Kuroda T., Doya K., Toyama K., Kimura M., Samejima K., Imamizu
H., Kawato M. (2001). fMRI study of human brain activity during reinforcement
learning. 31st Annual Meeting, Society for Neuroscience, San Diego, USA.
Hikosaka O., Nakahara H., Rand M.K., Sakai K., Lu X., Nakamura K., Miyachi S.,
Doya K. (1999). Parallel neural networks for learning sequential procedures.
Trends in Neurosciences, 22, 464-471.
Miyapuram K.P., Bapi R.S., Samejima K., Doya K. (2001). fMRI investigation of
the learning of visuo-motor sequences. 31st Annual Meeting, Society for
Neuroscience, San Diego, USA.
Nakahara H., Doya K., Hikosaka O. (2001). Parallel cortico-basal ganglia
mechanisms for acquisition and execution of visuo-motor sequences - A
computational approach. Journal of Cognitive Neuroscience, 13, 626-647.
Nakahara H., Doya K., Hikosaka O. (1998). Benefit of multiple representations
in parallel cortico-basal ganglia mechanisms for acquisition and execution of
visuo-motor sequences. International Basal Ganglia Society 6th Triennial
Meeting, 29.
Nakahara H., Doya K., Hikosaka O. (1998). Benefit of multiple representaitons
for motor sequence control in the basal ganglia loops. RIKEN Brain Science
Institute.
Nakahara H., Doya K., Hikosaka O., Nagano S. (1998). Reinforcement learningwith multiple representations in the basal ganglia loops for sequential motor
control. International Joint Conference on Neural Networks, 1553-1558.
Nakahara H., Doya K., Hikosaka O., Nagano S. (1997). Multiple representationsin the basal ganglia loops for acquisition and execution of sequential motor
control. 27th Annual Meeting, Society for Neuroscience, 23, 778.
Nakahara H., Doya K., Hikosaka O., Nagano S. (1997). Multiple representationsin the basal ganglia loops for sequential decision making. Technical Report of
IEICE, NC97-24.

Samejima K., Ueda Y., Doya K., Kimura M. (2003). Activity of striate projection
neurons encodes action-selective reward expectations. Society for
Neuroscience 33rd Annual Meeting, 78.
Samejima K., Ueda Y., Kimura M., Doya K., Schweighofer N. (2000).
Information coding of the striatal neurons during seqential movement. 30th
Annual Meeting, Society for Neuroscience.
Ueda Y., Samejima K., Doya K., Kimura M. (2003). Reward Value Dependent
Striate Neuron Activity of Monkey Performing Trial and Error Behavioral
Decision Task. Neuroscience Research, Vol. 46 Suppl. 1 S1-S220, S50.
Cortical Dynamics
Kawawaki, D., Shibata, T., Goda, N., Doya, K., Kawato, M. (2004). Anterior andsuperior lateral occipito-temporal cortex responsible for target motion
prediction during overt and covert visual pursuit. Society for Neuroscience
34th Annual Meeting, 64.
Matsumoto N., Okada M., Doya K., Sugase Y., Yamane S., Kawano K. (2001).
Dynamics of the face-responsive neurons in the temporal cortex.
Neuroscience Research, Supplement, 24, S73.
Okada M., Toya K., Kimoto T., Doya K. (1999). Retrieval dynamics ofassociative memory model can explain temporal dynamics of face-responsive
neurons in the IT cortex. 29th Annual Meeting, Society for Neuroscience,
Miami Beach, Florida, USA.
Tabata H., Shibata T., Taguchi S., Doya K., Kawato M. (2001). A simulation
study on smooth pursuit and ocular following responses based on an MST
neural-field model. Society for Neuroscience, San Diego, USA.
Bird SongDoya K., Sejnowski T.J. (1999). A computational model of avian song learning.
Gazzaniga MS, The New Cognitive Neurosciences, MIT Press, 469-482.

Doya K., Sejnowski T.J. (1998). A computational model of birdsong learning byauditory experience and auditory feedback. Brugge J, Poon P, Central
Auditory Processing and Neural Modeling, 77-88.
Doya K., Sejnowski T.J. (1995). A novel reinforcement model of birdsong
vocalization learning. Tesauro G, Touretzky DS, Leen TK, Advances in Neural
Information Processing Systems 7, MIT Press, 101-108.
Doya K., Sejnowski T.J. (1995). A computational model of birdsong
vocalization learning. Fourth IBRO World Congress of Neuroscience
Abstracts, 502.
Doya K., Sejnowski T.J. (1995). A model of birdsong vocalization learning.
Burrows M, Matheson T, Newland PL, Schuppe H, Nervous Systems andBehavior, 76.
Doya K., Sejnowski T.J. (1994). A computational model of song learning in theanterior forebrain pathway of the birdsong control system. 24th Annual
Meeting, Society for Neuroscience, 20, 166.
Neural Oscillators
Doya K., Boyle M.E.T., Beauchamp M., Selverston A.I. (1993). Computationalmodeling of the musculoskeletal system of the lobster gastric mill. 23rd
Annual Meeting, Society for Neuroscience, 19, 1602.
Doya K., Boyle M.E.T., Selverston A.I. (1993). Mapping between neural andphysical activities of the lobster gastric mill. Giles CL, Hanson SJ, Cowan JD,
Advances in Neural Information Processing Systems 5, Morgan Kaufmann,
913-920.
Doya K., Selverston A.I. (1994). Dimension reduction of biological neuronmodels by artificial neural networks. Neural Computation, 6, 696-717.
Doya K., Selverston A.I. (1993). A learning algorithm for Hodgkin-Huxley typeneuron models. Proceedings of 1993 International Joint Conference on
Neural Networks, 1108-1111.

Doya K., Selverston A.I., Rowat P.F. (1994). A Hodgkin-Huxley type neuronmodel that learns slow non-spike oscillation. Cowan JD, Tesauro G,
Alspector J, Advances in Neural Information Processing Systems 6, Morgan
Kaufmann, 566-573.
Schweighofer N., Doya K., Kawato M. (1998). A model of theelectrophysiological properties of the inferior olive neurons. The 5thInternational Conference on Neural Information Processing, 3, 1525-1528.
Recurrent Networks
Doya K. (2002). Recurrent neural networks: Supervised Learning. Arbib M,
The Handbook of Brain Theory and Neural Networks, Second Edition.
Doya K. (1995). Recurrent networks: Supervised learning. Arbib M, The
Handbook of Brain Theory and Neural Networks, (796-800)
Doya K. (1992). Bifurcations in the learning of recurrent neural networks.
Proceedings of 1992 IEEE International Symposium on Circuits and Systems,
2777-2780.
Doya K. (1991). A study of learning algorithms for continuous-time recurrent
neural networks. Department of Mathematical Engineering and Information
Physics, University of Tokyo.
Doya K. (1990). Learning temporal patterns in recurrent neural networks.
Proceedings of 1990 IEEE System, Man and Cybernetics Conference,
170-172.
Doya K., Yoshizawa S. (1992). Adaptive synchronization of neural andphysical oscillators. Moody JE, Hanson SJ, Lippmann RP, Advances in Neural
Information Processing Systems 4, 109-116.
Doya K., Yoshizawa S. (1991). Neural network model of temporal patternmemory. Systems and Computers in Japan, 22, 61-69.

Doya K., Yoshizawa S. (1991). Geometric analysis of the dynamics ofautocorrelation associative memory. International Conference on Artificial
Neural Networks, 1, 261-266.
Doya K., Yoshizawa S. (1990). Memorizing hierarchical temporal patterns inanalog neuron networks. Proceedings of 1990 International Joint Conference
on Neural Networks, San Diego, III:299-304.
Doya K., Yoshizawa S. (1989). Adaptive neural oscillator using
continuous-time back-propagation learning. Neural Networks, 2, 375-386.
Doya K., Yoshizawa S. (1989). Memorizing oscillatory patterns in the analog
neuron network. Proceedings of 1989 International Joint Conference on
Neural Networks, I:27-32.
Nakahara H., Doya K. (1996). Dynamics of attention as near saddle-nodebifurcation behavior. Touretzky DS, Mozer MC, Hasselmo ME, Advances in
Neural Information Processing Systems 8, MIT Press, 38-44.