## 4 Experiment and Results
### Q-Network Algorithm
#### 4.2.3 Algorithm Description  
First, we need to load the fall module to detect if the robot has fallen (see Figure 14). Every time the simulated robot fell down, we needed to restore the simulation manually because the simulated robot could not rise by itself. For this reason, we could not train the Q-network in one run; however, we needed to keep training the network just after the last run. For this reason, we needed to save the network model at the end of each run and to load it at the beginning of each new run. We needed to initialize the NAO proxies, the motion proxy that allows to send signals to the NAO motors; the posture proxy that allows one to set the robot in a \"home position\"; the sonar proxy, which tells us the information of the ultrasonic sensors; and the memory proxy to access the data recorded, v. gr. the angle of the joints. After that, the robot required activ  
Figure 12: NAO Inertial Unit. Figure 13: Fall Module Multi-Layer Perceptron (MLP). setting the stiffness of the body to 1--this was achieved using the instruction motion.setStiffnesses (\"Body\", 1.0). We initialized the number of _epochs_ to 50. Notice that we need many more iterations due to the fact that when the robot falls down, it cannot always stand up by itself, and the simulation has to be restored manually. For instance, imagine we set the number of epochs to 2000 and the robot fell down in the second iteration, then we would have to wait until the 2000 th iteration to restore the simulation, which would take considerably more time. Afterwards, we initialized gamma to 0.9, which means experience is considered, the closer gamma to zero, the less we take into consideration the experience of previous trials. In Reference [31], the authors used a technique to let the algorithm converge via _experience replay_. It works in the following way: During the run of the algorithm, all the experiences < s\({}_{t}\), a, r, s\({}_{t+1}\) > are stored in a _replay memory_. When training the network, random mini batches from the replay memory are used instead of the most recent transition. Before training we needed to define a variable for this purpose with _replay_ as an empty list; however, as we were running the algorithm multiple times, in our case it is not an empty list, but it was the list previously filled in a previous run, i.e., _replay_ = _lastreplay_. We opened a loop from \(i\) = 0 until \(i\) = epochs, set the robot in the \"home position,\" which was the NAO posture \"StandInit,\" then we read sonars with the memory proxy and stored it in _sonar_. We then read the _state_; remember that the state in the first level will be a vector of 4 values while in the second level is a vector of 11 values. Then, we opened a while loop with the condition that the NAO had not reached a terminal state. A terminal state was when the robot fell or when it reached the goal; that is, in the first level a ZMP was located in a specific interval, and in the second level, a distance of 8 cm was covered. Later, we ran the Q-network forward and stored the result in _qval_ (a vector). Suppose that we have the vector (1.2, \(-\)0.36, 2.2, 0.0). The action\({}_{t+1}\) will be the action 3 because the third value is the greatest; however, we wanted to explore more options in order to not fall in a local minimum, so with a probability of _epsilon_, we chose between the max action or a random action. This epsilon gradually decreased in such a way that after many iterations it became 0. Then, we applied the action\({}_{t+1}\) and observed the _reward_ and the _new state_, where the values of the reward were described in the previous section. At this point, we had [_state_, _action_, _reward_, _new state_]. We needed to store this tuple in _replay_ and repeat the process until the length of replay was the same as _buffer_; we set buffer as 60. Once the buffer was filled, we got a mini batch, which was a random sample of length 30 from the replay array. Thereafter, we looped over each element of the mini batch.