## 3 Methodological Approach  
According to [PERSON] and [PERSON] [23], a reinforcement learning solution to a control problem is defined as a finite horizon Markov Decision Process (MDP). At each discrete time-step \(t\), the agent observes the current state of the environment \(s_{t}\in S\), takes an action \(a_{t}\in A(s_{t})\), receives a reward \(r\):\(S\times A\rightarrow\mathbb{R}\) and observes the new state of the environment \(s_{t+1}\). At each episode of \(T\) time-steps, the environment and the agent are reset to their initial poses. The goal of the agent is to find a policy, deterministic \(\pi_{\theta}(s)\) or stochastic \(\pi_{\theta}(a|s)\), parameterised by \(\theta\) under which the expected reward is maximised. Traditional reinforcement learning algorithms are able to deal with discrete state and action spaces but, in robotic tasks, where both state and action spaces are continuous, discretising those spaces does not work well. Nevertheless, most state-of-the-art algorithms use deep neural networks to map observations with low-level control actions to be able to deal with continuous spaces. In our approach, a DRL algorithm is used to learn where to place the mobile manipulator to make a correct picking trial through the interaction with the environment. Those DRL algorithms need a huge amount of experience to be able to learn complex robotic behaviours and, thus, it is infeasible to train them acquiring experience in real world. In addition, the actions taken by the robot in the initial learning iterations are nearly random and both the robot and the environment might end up damaged as a result. Therefore, DRL algorithms are usually trained in simulation. Learning in simulation enables a faster experience acquisition and avoids material costs. Our study used a simulation-based approach, and we based it on open source robotic tools such as Gazebo simulator, Robot Operating System (ROS) middleware [24], OpenAI Baselines DRL library [25] and Gym/Gym-gazebo toolboxes [26; 27]. DRL algorithms follow value-based, policy search or actor-critic architectures [28]. Value based-algorithms estimate the expected reward of an state or state-action pair and are able to deal with discrete action spaces, typically using greedy policies. To be able to deal with continuous action spaces, policy search methods use a parameterised policy and do not need a value function. Usually, those methods have the difficulty of not easily being able to evaluate the policy and they have high variance. Actor-critic architecture is the most used one in the state-of-the-art algorithms and combines the benefits of the two previous algorithm types. On the one hand, actor-critic-based methods use a parameterised policy (actor), and, on the other hand, use a value or action-value function that evaluates the quality of the policy (critic). The proposed approach follows the actor-critic architecture. On the one hand, a parameterised policy \(\pi_{\theta}\) is used to be able to deal with both continuous state and action spaces in stochastic environments, encoded by the parameter vector \(\theta\). On the other hand, a parameterised value function is used to estimate the expected reward at each state or state-action pair, where \(w\) is the parameter vector that encodes the critic. The state value function \(V_{w}(s)\) estimates the expected average reward of all actions in the state \(s\). The action-value function \(Q_{w}(s,a)\), instead, estimates the expected reward of executing action \(a\) in state \(s\). Then, the critic's information is used to update both actor and critic. In DRL algorithms, both actor and critic are parameterised by deep neural networks, and the goal is to optimise those parameters to get the agent's optimal behaviour. In addition, DRL algorithms are divided into two groups, on-policy and off-policy, depending on how they are able to acquire experience. On-policy algorithms expect that the experience used to optimise their behaviour policy is generated by the same policy.