## 2 Problem Statement
### Reinforcement Learning for Mobile Robot Path Planning
The discrete-time state transition equations become the following:  
\[x_{t+1} = x_{t}+v_{t}\cos\theta_{t}\Delta t, \tag{8}\] \[y_{t+1} = y_{t}+v_{t}\sin\theta_{t}\Delta t,\] (9) \[\theta_{t+1} = \theta_{t}+\omega_{t}\Delta t,\] (10) \[v_{t+1} = v_{t}+\left(-\frac{1}{\tau_{v}}(v_{t}-u_{v})+\xi_{v}\right) \Delta t,\] (11) \[\omega_{t+1} = \omega_{t}+\left(-\frac{1}{\tau_{\omega}}(\omega_{t}-u_{\omega}) +\xi_{\omega}\right)\Delta t. \tag{12}\]  
The robot operates in an environment containing obstacles and goals, which influence the state transition. For example,  
* If the robot's position \((x_{t+1},y_{t+1})\) lies within an obstacle region, the state transition is interrupted, and the robot enters a collision state with a penalty reward. * Sensor noise can introduce uncertainty in the perceived state, modeled as an additive Gaussian noise \(\eta\sim\mathcal{N}(0,\Sigma)\) applied to the observed position and orientation: \[\tilde{s}_{t+1}=s_{t+1}+\eta,\] (13) where \(\tilde{s}_{t+1}\) is the observed state (assuming that the sensing mechanism includes a complete GPS-INS). For the differential-drive mobile robot example, the deterministic component of the state transition is governed by the discretized nonlinear dynamics described earlier. However, to account for stochastic effects in the real-world environment, the state transition probability density can be expressed as follows:  
\[\mathcal{P}(s^{\prime}|s,a)=\mathcal{N}(s^{\prime};f(s,a),\Sigma), \tag{14}\]  
where  
* \(f(s,a)\) is the deterministic state transition function derived from the discretized equations of motion, \[f(s,a)= \begin{bmatrix}x_{t}+v_{t}\cos\theta_{t}\Delta t\\ y_{t}+v_{t}\sin\theta_{t}\Delta t\\ \theta_{t}+\omega_{t}\Delta t\\ v_{t}+\left(-\frac{1}{\tau_{v}}(v_{t}-u_{v})+\xi_{v}\right)\Delta t\\ \omega_{t}+\left(-\frac{1}{\tau_{\omega}}(\omega_{t}-u_{\omega})+\xi_{\omega} \right)\Delta t\end{bmatrix} ,\] (15)* \(\Sigma\) is the covariance matrix representing the stochastic uncertainties in the state transition, which could arise from sensor noise, actuator variability, or environmental factors. The covariance matrix \(\Sigma\) is often diagonal under the assumption that the noise in different state dimensions is uncorrelated. For instance,  
\[\Sigma=\text{diag}(\sigma_{x}^{2},\sigma_{y}^{2},\sigma_{\theta}^{2},\sigma_{v} ^{2},\sigma_{\omega}^{2}), \tag{16}\]  
where \(\sigma_{x}^{2},\sigma_{y}^{2}\) represent the positional uncertainties, \(\sigma_{\theta}^{2}\) the orientation uncertainty, and \(\sigma_{v}^{2},\sigma_{\omega}^{2}\) the variances in the linear and angular velocities due to stochastic disturbances. The environment in which the robot operates significantly affects the state transition function. In the presence of obstacles or terrain constraints, the deterministic dynamics \(f(s,a)\) may no longer accurately describe the transition and \(\mathcal{P}(s^{\prime}|s,a)\) must be adjusted accordingly. For example,  
* **Obstacle Collisions:** If the predicted state \(s^{\prime}=(x^{\prime},y^{\prime},\theta^{\prime},v^{\prime},\omega^{\prime})^ {\top}\) lies within an obstacle region, the state transition function could assign zero probability to this transition and redirect the robot to a collision state \(s_{\text{collision}}\). Mathematically, \[\mathcal{P}(s^{\prime}|s,a)= \begin{cases}\mathcal{N}(s^{\prime};f(s,a),\Sigma ),&\text{if }(x^{\prime},y^{\prime})\
otin\mathcal{O},\\ \delta(s^{\prime}-s_{\text{collision}}),&\text{if }(x^{\prime},y^{\prime})\in \mathcal{O},\end{cases} \] (17) where \(\mathcal{O}\) is the set of obstacle regions, and \(\delta(\cdot)\) is the Dirac delta function centered at the collision state. * **Dynamic Obstacles:** In environments with moving obstacles, the state transition must account for dynamic constraints imposed by the obstacle's trajectory.