Patent Document ID: 8521678
Application ID: 12792853
Patent Flag: 1

Claim One:
1. An apparatus having a memory section and a learning control system, the learning control system performing learning of action values of actions of the apparatus which identifies its state as one of predetermined states, and selects an action based on the obtained action values and the identified state, the learning control system comprising: n action value learning devices including the first to the n th learning devices which perform, using a computer and the memory section, learning of n action values from Q1 to Qn, respectively, assuming that n is a positive integer; and an action value determining device which determines, using the computer and the memory section, the total action value of an action Q of each state based on the n action values of the n action value learning devices, wherein the first target value of the first action value learning device is determined based on the reward r that is obtained after an action has been carried out by the next state and a total action value Q′ that was prepared for the action selection in the next state, and the first learning device retrieves from the memory section and updates the first action value Q1 using the first target value, wherein when n is 2 or more, the n-th target value of the n th action value learning device is set to the difference between the (n−1) th target value of the (n−1) th learning device and the action value Qn−1, and the n th learning device retrieves from the memory section and updates the n th action value Qn using the n th target value, and wherein a learning coefficient, the n th target value and an update amount of action value Qn are represented respectively as αn, Tn and ΔQn, and the following expressions are held when n is two or more, Δ ⁢ ⁢ Q n = α n ⁡ ( T n - A n ⁢ Q n ) T n = ( 1 - α n - 1 ) ⁢ ( T n - 1 - A n - 1 ⁢ Q n - 1 ) A n = ∑ p n ⁢ α p ⁢ ∏ q = 1 p - 1 ⁢ ( 1 - α q ) α n wherein An is a coefficient for correcting Qn.