# Reward Function and Configuration Parameters in Machine Learning of a Four-Legged Walking Robot
## 2 Materials and Methods
### ML-Agents Module
The most important values are as follows:  
* hyperparameters batch_size--This is the number of data in each iteration. This value should always be much smaller than the buffer size. If continuous actions are used, it takes a value of 100 s. For discrete actions, it takes a value of 10 s;
* hyperparameters buffer_size--This is the number of data that needs to be collected before updating the model. The buffer size should be much larger than batch_size. Typically, a larger buffer results in more stable model updates during the training process (default = 10,240);
* hyperparameters learning_rate--This is the initial learning rate for the simple gradient method. This value corresponds to the impact of each update step. If the training is unstable or the reward value does not increase consistently, it is probably set too high (default = 3
* hyperparameters beta--This is the effect of the regularisation of entropy, which makes the agent's policy more random. Owing to this parameter, agents can properly explore the action space during training. The higher the value, the more random the agent's actions. This parameter should be adjusted so that the entropy value slowly decreases as the reward increases (default = 5
* hyperparameters epsilon--This affects how quickly the agent's policy evolves during the learning process. A small value produces more stable updates between episodes but slows the learning process (default = 0.2);
* hyperparameters lambda--This is the learning regularisation parameter. It provides a trade-off between the set and actual reward values (default = 0.95);* hyperparameters num_epoch--This is the number of epochs that pass through the buffer during the optimisation (default = 3);
* hyperparameters learning_rate_schedule--This determines how the rate of learning changes over time. For PPO, it is recommended to reduce the learning rate to max_steps to make the process more stable (default = linear);
* Network_settings normalises--This is a function that is applied to the vector input. It is useful for complex continuous control problems, but can be harmful for simpler discrete control problems (default = disabled);
* Network_settings hidden_units--This is the number of units in the hidden layers of the neural network. For simple combinations of input data (observations), it should take a small value (default = 512);
* Network_settings num_layers--This is the number of layers hidden in the neural network. For simple problems, fewer layers can have a faster learning rate and better performance. More layers may be useful for more complex problems (default = 3);
* Network_settings vis_encode_type--encoder type for encoding visual observations (default = simple)Extrinsic strength--This is the coefficient by which the award granted by the environment should be multiplied. Its value depends heavily on the reward signal (default = 1.0);
* Extrinsic gamma--This is the coefficient used to calculate successive rewards from the environment. It determines how much the agent should consider possible rewards received in the future;
* keep_checkpoints--This is the maximum number of model checkpoints to keep. The checkpoints are written after the number of steps is specified in the checkpoint_interval variable (default = 500,000). When the maximum number of records is reached, the oldest checkpoint is deleted when saving a new point (default = 5);
* max_steps--This is the total number of steps that the agent must take in the environment before the learning process ends. If multiple agents are trained simultaneously, all the steps they take will affect this number (default = 5,000,000);
* time_horizon--This defines how many steps the agent must take before adding data to the buffer. When this limit is not reached, the estimated value is used to predict the expected value. This number should be large enough to capture all the important agent behaviours in a given episode (default = 64);
* summary_freq--This is the number of data that needs to be collected before training statistics are generated and displayed.