## 2 Problem Statement
### Reinforcement Learning for Mobile Robot Path Planning  
In the context of mobile robot path planning, the goal is to compute a policy that enables the robot to navigate from a starting position to a target position while avoiding obstacles and minimizing path costs. The MDP formulation of this problem is described as follows:  
* State Space (\(\mathcal{S}\)): Each state \(s\in\mathcal{S}\) represents the robot's current status and environmental context. For example, \(s\) may include the robot's position \((x,y)\), orientation \(\theta\), velocities, and potentially sensor measurements, such as distances to nearby obstacles and the relative position of the goal. * Action Space (\(\mathcal{A}\)): The action \(a\in\mathcal{A}\) represents a control input applied to the robot. For instance, \(a\) could define discrete motion commands (e.g., \(\mathsf{MoveForward}\), \(\mathsf{TurnLeft}\)), or continuous control signals such as linear velocity \(v\) and angular velocity \(\omega\). * State Transition (\(\mathcal{P}(s^{\prime}|s,a)\)): This models the robot's motion dynamics and environmental interactions. For example, if the robot is controlled using differential drive kinematics, the transition from \(s\) to \(s^{\prime}\) after applying \(a\) depends on equations of motion. Stochasticity may arise due to wheel slip, sensor noise, or dynamic obstacles. * Reward Function (\(r(s,a)\)): The reward function incentivizes behaviors that achieve the desired goal while avoiding undesirable outcomes. * Discount Factor (\(\gamma\)): The discount factor determines how much emphasis is placed on immediate versus long-term navigation performance. For instance, \(\gamma\) close to 1 ensures that the robot considers the entire trajectory to the goal. Using reinforcement learning, the agent iteratively interacts with the environment to learn an optimal policy \(\pi^{*}\) that selects actions to navigate effectively while avoiding obstacles. State-of-the-art deep reinforcement learning algorithms such as Proximal Policy Optimization (PPO) [4] and Soft Actor-Critic (SAC) [2] are particularly suited for this task, as they handle high-dimensional state-action spaces and continuous control effectively. Problem Formulation for a Differential Drive Mobile Robot:  
For a differential-drive mobile robot, the state \(s\in\mathcal{S}\) is defined as follows: \(s=[x\ y\ \theta\ \
u\ \omega]^{\top}\), where \(x,y\) are the Cartesian coordinates of the robot's position in a 2D plane, \(\theta\) is the orientation (heading angle) of the robot relative to the global frame, \(v\) is the linear velocity of the robot, and \(\omega\) is the angular velocity of the robot. The control input \(a\in\mathcal{A}\) is defined as \(a=[u_{v}\;u_{\omega}]^{\top}\), where \(u_{v}\) and \(u_{\omega}\) are the commanded linear and angular velocity inputs, respectively. The robot's motion dynamics can be described using the following system of nonlinear differential equations:  
\[ \begin{array}{rclrclrclrcl}\dot{x}&=&v\cos\theta,&&\dot{y}&=&v\sin\theta,&& \dot{\theta}&=&\omega,\\ \dot{v}&=&-\frac{1}{\tau_{v}}(v-u_{v})+\xi_{v},&&\dot{\omega}&=&-\frac{1}{\tau _{\omega}}(\omega-u_{\omega})+\xi_{\omega},&&\end{array}  \tag{7}\]  
where \(\tau_{v},\tau_{\omega}\) are the time constants for the linear and angular velocity dynamics, and \(\xi_{v},\xi_{\omega}\), are the stochastic disturbances representing noise in the robot's actuators, modeled as Gaussian random variables, \(\xi_{v}\sim\mathcal{N}(0,\sigma_{v}^{2})\) and \(\xi_{\omega}\sim\mathcal{N}(0,\sigma_{\omega}^{2})\). For practical implementation in reinforcement learning, the continuous time dynamics are discretized using a time step \(\Delta t\).