## 4 Methodology
### Background: Reinforcement Learning
#### 4.1.2 Model Architecture  
Since MO-MPO is an actor-critic algorithm, this section details the neural network architectures used for both the policy (actor) and the value estimation (critic) networks. Determining the number of layers and neurons per layer constitutes a key hyperparameter choice that involves balancing model complexity with the capacity to capture meaningful patterns in the data. Following the architectural guidelines recommended by [PERSON] et al. [29], we adopt a configuration consisting of three fully connected hidden layers. This architecture offers a practical trade-off between expressiveness and computational efficiency, making it well-suited for continuous control tasks. **Actor Network:** The actor network \(\pi\) comprises three fully connected hidden layers, each with 256 neurons. This architecture strikes a well-established balance between depth and width, and aligns with the default configuration of the Soft Actor-Critic algorithm as implemented in Stable-Baselines3 [30]. The choice of 256 neurons per layer is motivated not only by its empirical effectiveness, but also by its computational efficiency: it matches the 32-thread CUDA warp size, thereby enhancing GPU utilization and kernel occupancy [31; 32]. This specific 3 \(\times\) 256 configuration is also recommended in the Acme framework's implementation of MO-MPO by [PERSON] et al. [29], where it is applied to both the actor and critic networks. The input dimension corresponds to the number of controllable parameters (four in this case). The network produces two output layers, a mean vector \(\mu\) and a covariance matrix \(\Sigma\), which together define the parameters of the policy distribution. The actor is trained using a two-step EM-style procedure:  
E-step: Constructs a non-parametric action distribution \(q(a\mid s)\) that is weighted by a prioritized combination of Q-functions, using weights \(\alpha_{k}\) for each objective. \[q(a\mid s)\propto\exp\left(\frac{1}{\eta}\sum_{k=1}^{N}\alpha_{k}Q_{k}^{\pi}( s,a)\right) \tag{2}\]  
M-step: Updates the policy \(\pi_{\theta}(a\mid s)\) by minimizing the KL divergence between the new action distribution \(q(a\mid s)\) and the current policy. \[\mathcal{L}_{\text{actor}}=\mathbb{E}_{s\sim\mathcal{D}}[\text{KL}(q(a\mid s) \parallel\pi_{\theta}(a\mid s))] \tag{3}\]  
* \(\pi_{\theta}(a\mid s)\): Parametric actor policy. * \(q(a\mid s)\): Target distribution computed from the weighted sum of Q-functions. * \(\text{KL}(q\parallel\pi)\): KL divergence between the target and the current policy. * \(\mathbb{E}_{s\sim\mathcal{D}}\) represents the expected value over the state distribution \(s\)\(\sim\)\(\mathcal{D}\), where \(\mathcal{D}\) denotes the dataset containing sampled states. * \(\mathcal{L}_{\text{actor}}\) denotes the loss function. Critic Network: Similar to the actor network, the critic network contains three fully connected hidden layers with 256 neurons per layer. As each objective requires its own \(Q\)-function, the total number of critic networks equals the number of objectives, two in this implementation. Both the state and action spaces determine the input dimension. Batches of state-action pairs are sampled from the replay buffer and passed to the critic, which returns a scalar \(Q\)-value for each objective. The critic's primary role is to evaluate actions by approximating their expected returns, a process formally referred to as policy evaluation. The MO-MPO algorithm incorporates a technique known as retracing, wherein state-action pairs are recomputed using the updated policy parameters. This mechanism is applied across all objectives and can be extended with a prioritization strategy tailored formulti-objective settings [33]. As described by [PERSON] et al.