# Reward Function and Configuration Parameters in Machine Learning of a Four-Legged Walking Robot
## 4 Discussion  
The first of the studies conducted focused on examining the impact of the reward function on the course of the reinforcement learning process. Owing to the differences in the absolute value of this function for each of the learning cycles, it is difficult to compare them directly; however, some general dependencies can be clearly seen in the graphs that compare the characteristic values of the process. During the first cycle, when parts of the reward function for achieving the goal and the robot's gait were the highest, the worst quality of learning is noticeable. The probable reason for this is that the fewest points are awarded to the agent for its actions. Too little motivation to achieve the goal also means that the longest time had to pass before the total reward began to grow rapidly. During the second cycle, when the reward for achieving the goal is doubled, the beginning of effective learning is seen as the fastest of all the cases analysed. The downside of this solution is that such a heavy emphasis on achieving the goal minimises the robot's efforts to increase the reward function from other sources. This shows a much gentler slope of the cumulative reward curve at the end of the learning process. The third cycle emphasised the acquisition of the reward function while walking to a greater extent compared to the previous cycles. This translated into a more delayed start towards achieving the goal but significantly decreased the final number of steps in the episode. Furthermore, the steep slope of the cumulative reward curve at the end of the learning process suggests that if it continues, the agent would be willing to further improve its score, in contrast to the situation in the second cycle. The values adopted in the third cycle were characterised as the best among those considered; and, on that basis, the second study was conducted. It, again, consisted of three cycles, each of which examined the influence of a different hyperparameter on the course of the learning process. In the first cycle, the default value of the gamma parameter was changed from 0.995 to 0.8. This parameter, according to the documentation of the model that was used, is responsible for how far into the future the agent should go in the context of a potential reward. Typically, this parameter should have large values if the value of the reward function is to increase in the future; and, if the value increases immediately, gamma may be smaller. Of the three modifications analysed in the configuration file, this change, as well as increasing the reward for achieving the goal, led to the fastest increase in the reward function. However, from the perspective of the entire study, this modification resulted in the lowest cumulative value of the reward function and, at the same time, the most steps necessary for the agent to be able to achieve the goal. In the second cycle, the number of hidden layers in the learnt neural network was reduced from three to one. In the case of this hyperparameter, it should be expected that the more complicated the learning environment, the more layers are needed for effective learning. On this basis, we can conclude that the created environment was interpreted by the trainer as not very complex because reducing the number of layers was the best modification among the analysed configuration file changes. Although the agent reached the goal for the first time after a longer time than in the case of reducing the gamma value, the number of steps needed for this and the value of the reward function itself quickly began to reach much more favourable values than those in the other cases that were analysed. The last tested configuration was the change in the number of units in each hidden layer from 512 to 32. Numerous units are recommended in environments where there are many interactions between variables representing observations.