## 2 Materials and Methods
### Reinforcement Learning in Continuous Environments
### Dynamic Movement Primitives (DMP)

Since the previously presented model for learning from demonstrations is based on the analysis of coordinate systems in the operational (Cartesian) system of the robot, this section will introduce a trajectory model used to approximate Cartesian states, which will be further enhanced by a modification that couples this model with the proposed analysis of coordinate system activities [16]. The basis of this trajectory model is the primitive motion model--DMP. Specifically, a special variation of the DMP model capable of approximating states defined in the Cartesian coordinate system is utilized. The forcing term is replaced by establishing interdependence between states and time. This is achieved by recalculating the current goal of the DMP dynamic system at each time step according to a time-dependent probabilistic function of predefined states. This approach was originally presented in Ref. [1] to encode stiffness information into the trajectory for robots controlled by a given torque. The stiffness matrix mentioned is not used for this approach. The system's acceleration is calculated as the sum of \(N\) proportional differential systems, where \(N\) denotes the number of states, \(y\) and \(\dot{y}\) represent the current position and velocity of the system, and \(a_{y}\) and \(b_{y}\) are gain parameters. \[\tau^{2}\overset{\wedge}{y}=\sum_{i=1}^{K}h_{i}(t)\cdot a_{y}\big{[}b_{y}(\mu_ {i}-y)-\tau\dot{y}\big{]}, \tag{1}\]

The aforementioned time dependence is achieved by using an exponentially decaying function \(t=-\alpha_{x}\cdot t\) as a function of time, where the variable \(t\) is initialized with \(t=1\) at the beginning of each motion and gradually converges to zero at each time step. The rate of decay, i.e., convergence of this function to zero, is defined by the gain parameter \(\alpha_{x}\)[17]. This temporal function is linked to the system dynamics through a set of \(N\) normal distributions \(\mathcal{N}\left(t;\mu_{i}^{t},\Sigma_{i}\right)\) with uniformly distributed mean values of distributions from zero to one, where each individual distribution represents one state from the state vector \(\mu\). The time-dependent probability of the system being in a certain state is given in Equation (2). \[h_{i}(t)=\frac{\mathcal{N}\big{(}t;\mu_{i}^{t},\Sigma_{i}\big{)}}{\sum_{k=1}^{ K}\ \mathcal{N}(t;\mu_{k}^{t},\Sigma_{k})}, \tag{2}\]

The implemented DMP based trajectory planning algorithm is shown in Algorithm 2. The algorithmic task consists of converting the initial trajectory into a DMP record. Using iterative search, an attempt is made to find a trajectory that passes through the given spatial position X, which was not previously part of the initial trajectory. Using this model, sequential tracking of arbitrary states defined either in the operational (Cartesian) or configuration system of the robot can be achieved. Furthermore, it can be observed how the shape of the exponentially decaying time function determines how much time is \"allocated\" to a specific state within the state vector [18]. Using the standard form of this function, it is evident that the function has a higher rate of decay at the beginning of the function (after initialization with a value of 1), indicating that states appearing in this time period are less important than states appearing at the end of the function, where the rate of time decay is much smaller as the function converges to a value of zero. ### Neural Network Architecture

The input layer receives a 25-dimensional vector representing the state of the environment. This could include variables such as joint angles, joint velocities, distances to obstacles, or other environmental factors that are relevant for the task. The number of nodes in the first hidden layer is 64. Using the ReLU (rectified linear unit) activation function, the network introduces non-linearity, allowing the model to learn complex patterns in the input data. The number of nodes in the second hidden layer is 64, and the activation function is also ReLU.