Quick Online Adaptation with Reinforcement of Simulated Soccer Agents

This work describes an adaptive system that uses a counterpropagation artificial neural network in decision-making. The output layer of the network is trained by a specially designed one-step reinforcement learning algorithm. This approach allows creating an agent that can quickly adapt to the surrounding environment on the basis of the trial-and-error method which, in turn, enables one to use it for a real-time adaptation in the RoboCup simulation environment. Introduction A huge number of states (108×50) and possible policies 300 within one cycle [1] makes the RoboCup simulation environment [2] extremely popular for different heuristic learning systems. Although the RoboCup environment allows using real-time (online) learning, nevertheless the suppressing majority of RoboCup agents use offline training as a basis for decision-making. This means that the agents are trained to react to every situation determined and do not change their policy of behavior in the process of a game. However, it is highly possible that during the game against a given opponent, it will found out that the behavior of the agent is not correct in some situations. In this case, the online adjustment of this behavior is needed. Moreover, (1) the number of iterations necessary for retraining should be minimal; and (2) the agent should manage in time, which is given him for decision-making on each game-cycle (usually 100 ms). This paper proposes to use a counterpropagation artificial neural network [3] with specially designed quick learning algorithm with reinforcement, as one of the possible ways of implementation of online adaptation. Counterpropagation artificial neural network Counterpropagation neural network consists of two layers of neurons: a Kohonen selforganizing feature map and a Grossberg star. The layer of the Kohonen neurons allows creating classes of input patterns and then relates a newly presented pattern to 2 Alexey Vasilyev, Andrei Kapishnikov, and Anatoly Sukov one of the classes. The layer of the Grossberg neurons is used for forming the required reaction to each pattern. The training of the counterpropagation neural network usually consists of two stages. During the first stage the Kohonen layer is trained – the input patterns are divided into groups based on their similarity. This stage involves unsupervised training. During the second stage, the Grossberg layer is trained with supervision using the already formed classes. The self-organizing Kohonen layer gives outputs in non-deterministic positions, which are mapped into the desired outputs by the Grossberg layer. Online counterpropagation neural network reinforcement training There are such situations when we only know about an executed action – whether it was correct or not, and, in case of incorrect action, we do not know which action had to be executed instead. In such cases, it is necessary to use a learning algorithm with reinforcement [4], which is based on observation of agent’s actions. Depending on the results, the agent is rewarded or punished, without explaining to the trained object which actions were correct. In this case, the surrounding environment can play the role of the supervisor. Fig. 1. The counterpropagation neural network with a reinforcement algorithm. Here r – the reinforcement signal, which is given to the Grossberg layer; x1, . , xn – network inputs, y1, . , ym – network outputs As it was mentioned above, the training of the counterpropagation neural network is supervised. In order to introduce the reinforcement into the neural network, it is necessary to modify the training algorithm so that the signal of reinforcement r would be taken into account (see Figure 1). Quick Online Adaptation with Reinforcement of Simulated Soccer Agents 3 Alternative reinforcement for two actions The main point of alternative reinforcement for two actions lies in the fact that if an agent can execute only two possible actions a1 and a2, knowing the action, which was executed erroneously, it is possible to say, what action should be taken instead of it. In this case, the weights of the neurons of the Grossberg layer can be slightly adjusted according to the following formula: , , i j i j j w w s c r x = + ⋅ ⋅ ⋅ , (1) where wi,j is the weight of the i-th input of the j-th neuron; s = +1 if yj approaching a maximally possible value and s = -1 otherwise; (0.1) c∈ – learning rate; r – reinforcement signal; {0,1}, j x j = ∀ – input vector, which comes to the Grossberg layer from the Kohonen one. Since the Grossberg neurons do not have an activation function, the situation is possible when the output value of the neuron is outside of the allowed range. To avoid this situation, we have introduced parameters vmin and vmax that correspond to the minimal and maximal value of the neuron output, respectively. Then equation (1) looks as follows: ( ) , , min max min max( , ), i j i j j w w s c r x v v = + ⋅ ⋅ ⋅ . (2) Alternative reinforcement for many actions The question arises for a more general case, when the number of possible actions is more than two: if the action performed by the system proved to be incorrect then what alternative should be used instead in the same situation next time. The majority of learning methods solve this problem by introducing randomization into the process of decision-making that leads both to the probabilistic selection of actions and simultaneously exploration of the most promising areas of the search space for all state-action pairs. The obvious disadvantage of this approach is the long period of learning. To overcome this shortcoming, we suggest including reinforcement with priorities into the decision-making process. Another alternative is employment of reinforcement based on the preceding experience. Alternative reinforcement with priorities In order to use reinforcement with priorities, it is necessary to assign the vector, which defines the priorities of all possible actions in the specific game situation. The value of the vector determines the distribution of the positive reinforcement signal between alternative actions if the action recently executed proved to be incorrect. The values of vector elements are assigned by an expert. For example, let us consider the case when an agent who plays the role of a forward can be found with the ball near the opponent’s goal. We assume that the agent can take one of four actions in that case, namely: 1) to pass, 2) to outplay the opponent’s agent, 3) to strike the ball into 4 Alexey Vasilyev, Andrei Kapishnikov, and Anatoly Sukov the goal, 4) to take up a more advantageous position for the strike or pass; accordingly, we have the vector P of four elements p1, p2, p3 and p4. For example, after analysis of games with opponent’s team participation, the expert can decide that the most preferable action is to strike the ball into the goal, because the goalkeeper of the opponent’s team badly intercepts the ball; accordingly, the value of the p3 element will be maximal. The greatest p3 value does not mean that the agent will constantly make the decision ‘to strike the ball into the goal’ but only that the weight of this decision will grow more rapidly in case of unsuccessful execution of another alternative. The following formula determines the distribution of the positive reinforcement signal: