## 2 Methods and Data Acquisition for Learning
### Dimensionality Reduction of Policy Parameter Space with Autoencoders  
Let us consider parametric policies \(\Pi(\mathbf{\theta})\), where \(\mathbf{\theta}\in\mathbf{\Theta}\) are the policy parameters and \(\mathbf{\Theta}\subset\mathbb{R}^{d}\) is a \(d\) dimensional space formed by all valid policy parameters \(\mathbf{\theta}\). [PERSON] and [PERSON] [15] noted that the dimensionality of \(\mathbf{\Theta}\) can be problematic when learning optimal policies for a given task. Intuitively, learning should become easier and faster to perform if we could implement it in a lower-dimensional parametric space. For this purpose, we must first embed all valid policies for the given task into a lower-dimensional parametric space. This is often possible because the space of all valid policies for the given task forms a low-dimensional manifold in the full parameter space \(\mathbf{\Theta}\)[9]. As shown in [30], deep autoencoder neural networks enable dimensionality reduction while retaining the most pertinent information in the low-dimensional policy representation. The autoencoder neural network is trained so that its input data matches its output data as precisely as possible. The input data are pushed through the network layers until they reach the bottleneck--the layer with the least number of neurons, also called the latent space. We denote the values of the neurons in the bottleneck layer by \(\mathbf{\theta}^{\text{AE}}\). The part from the input to the bottleneck is called the encoder part. The second part of the autoencoder, called the decoder, pushes data from the bottleneck through expanding layers, so that the outputs \(\widetilde{\mathbf{\theta}}\) of the autoencoder match the input data \(\mathbf{\theta}\) as closely as possible. Thus, the encoder and the decoder functions are nearly inverse \(\mathbf{F}_{\text{dec}}\approx\mathbf{F}_{\text{enc}}^{-1}\). The dimensionality of the data is now defined by the number of neurons in the bottleneck layer--the latent space (also referred to as code). Its dimensionality \(d_{\text{AE}}\) is usually significantly lower than the dimensionality of the original space, i.e., \(d_{\text{AE}}<d\). Thus, given an input data, the values of the neurons in the latent space are affected by the activation functions of the other hidden layers. These activation function can be, for example, the hyperbolic tangent (tanh), the sigmoid function, or a rectified linear unit [38]. For the tanh function, the output values of neurons at layer \(n\) are given by \(\mathbf{h}_{n}=\tanh(\mathbf{W}_{n}\mathbf{h}_{n-1}+\mathbf{b}_{n})\), where \(\mathbf{h}_{n-1}\) are the input values for the neurons at layer \(n\) and \(\mathbf{W}_{n}\), and \(\mathbf{b}_{n}\) are autoencoder parameters. Their values are determined by training. To train the AE network, that is to compute the appropriate autoencoder parameters \(\xi^{\star}\), we minimize  
\[\xi^{\star} = \arg\min_{\xi}\frac{1}{N_{s}}\sum_{i=1}^{N_{s}}L(\xi;\theta_{i}, \mathbf{F}_{\text{dec}}(\mathbf{F}_{\text{enc}}(\theta_{i})), \tag{1}\]  
where \(\xi=\bigcup_{n}\xi_{n}\), \(\xi_{n}=\{\mathbf{W}_{n},\mathbf{b}_{n}\}\), combines all the parameters defining the autoencoder network, \(\theta_{i}\) are the example policy parameters, \(L\) is the Euclidean distance between the input and output of the autoencoder network, and \(N_{s}\) is the number of samples, i.e., in our case this is the number of trajectories. We use stochastic gradient descent to train the network, thus the results of optimization depend on the initialization of the optimization process. For this reason, the network is trained several times using different initial parameters. The results are then averaged. The training of each autoencoder network is terminated if the results do not improve in 60 consecutive validation steps. This number was determined empirically.