## 4 Experiments and Result
### Training Process of RL Agent and Results of Experiments
#### 4.5.1 Experiment #1  
In this experiment, the distribution of semiproduct versions was assumed as a uniform distribution. The arrival rate distribution was assumed to be exponential with a scale parameter of 30 [s].  
Figure 10 shows the results obtained in the training process for the PPO and A2C algorithms. The graphs show the values of the average length of the episode (ep_len_mean) and the average reward per episode (ep_rew_mean). The PPO algorithm, for default parameter values, does not obtain better results during the training process. There is a visible increase in the reward obtained for the A2C algorithm (from 30 to 50). As the agent's observation of the environment and actions are taken after the end of processing for individual semi-finished products, the value for the length of the episode corresponds to the achieved system throughput for the assumed simulation time.  
In the next step, several training sessions were performed for different values of selected parameters of the PPO algorithm (which showed no improvement for the default parameters). The changes included the total number of samples to train on, the number of steps to run for each environment per update and the discount factor _gamma_.  
In Figure 11, the results obtained in the training process for the PPO are presented. The description shows only the values of the parameters that differ from the initial values. A significant improvement in the learning process can be seen.  
Table 2 synthetically shows the reward values obtained for selected parameters.  
\begin{table}
\begin{tabular}{c c c} \hline \hline
**Parameter** & **PPO** & **A2C** \\ \hline reward function & acc. (6) & acc. (6) \\ obs. space & 2 nd vector of (4) & 2 nd vector of (4) \\ action space & acc. (5) & acc. (5) \\ simulation time & 14400 t.u. & 14400 t.u. \\ \hline \hline \end{tabular}
\end{table}
Table 1: _Cont._  
Figure 10: Training process results for the initial parameter values of the PPO and A2C algorithms.  
As in the case of the PPO algorithm, the next step was devoted to training the A2C algorithm for different parameter values. As before, the changes included the total number of samples to train on, the number of steps per update, and the discount factor.  
In Figure 12, the results obtained in the training process for A2C are presented. No significant improvement in learning is observed.  
Then, the best-trained PPO and A2C models were used to control the sequence of semiproducts in the simulation environment. Figure 13a-d show the average results of the total reward for 20 replications obtained in FlexSim for the random allocation of semiproducts to the production cell, the FCFS/longest waiting time dispatching rule, the trained A2C model, and the trained PPO model.  
Figure 11: Training process results for various parameter values of the PPO algorithm: (**a**) _total_timesteps_ = 30,000; (**b**) _total_timesteps_ = 30,000, _n_steps_ = 512; (**c**) _total_timesteps_ = 30,000, _n_steps_ = 512, _gamma_ = 0.89; (**d**) _total_timesteps_ = 30,000, _n_steps_ = 512, _gamma_ = 0.69.  
Figure 12: Training process results for various parameter values of the A2C algorithm: (**a**) initial parameters; (**b**) _total_timesteps_ = 30,000, _n_steps_ = 5; (**c**) _total_timesteps_ = 20,000, _n_steps_ = 5, _gamma_ = 0.89; (**d**) _total_timesteps_ = 20,000, _n_steps_ = 2, _gamma_ = 0.89.  
\begin{table}
\begin{tabular}{c c c c} \hline \hline _total_timesteps_ & _n_steps_ & _gamma_ & **ep_rew_mean** \\ \hline
10,000 & 5 & 0.99 & 50.79 \\
30,000 & 5 & 0.99 & 50.77 \\
20,000 & 5 & 0.89 & 50.9 \\
20,000 & 2 & 0.89 & 52.06 \\ \hline \hline \end{tabular}
\end{table}
Table 3: A2C algorithm training results.