## 4 Proposed System for Patient-Robot Co-Navigation
### Shared Control in Crowded Environments Using Reinforcement Learning
#### 4.2.1 Shared Control Considering Only Dynamic Obstacles (Model V1)
Let \(\mu_{t}\) be the reward the robot has for transiting the state \(s_{t}\) to \(s^{t+1}\) and the \(\gamma\) be the discount factor, the total expected reward until the episode ends would be \(\mu=\sum_{k=0}^{\infty}(\gamma^{k}\mu_{k})\), where k is the total number of time steps. An episode includes all of the timesteps from the start until the robot reaches the goal. To ensure that the model learns a policy of shared control and crowd navigation, the proximal policy optimization (PPO) RL algorithm proposed by [PERSON] et al. [52] is used. the loss function used in PPO is a combination of the clipped surrogate objective and the entropy bonus as proposed by [PERSON] et al. The reward function \(\mu_{k}\) proposed by [PERSON] et al. [43] is modified to include the co-navigation objectives of our system, and it is as follows:  
\[\mu_{k}=\left\{ \begin{array}{ll}-20&d_{min}^{t}<0\\ -20&d_{p,r}^{t}>0.25\\ 2.5(d_{min}^{t}-0.25)&0<d_{min}^{t}<0.25\\ 10&d_{goal}^{t}<\rho_{robot}\\ 2(-d_{goal}^{t}+d_{goal}^{t-1})&otherwise\end{array} \right. \tag{5}\]  
At each time step (\(t\)), \(d_{min}^{t}\) represents the smallest possible distance between the robot and any individual who is not the patient (i.e., any human excluding the patient) at that time. The reward function is designed to incentivize the robot's behavior toward reaching its goal while avoiding collisions with humans and maintaining contact with the patient. The distance between the robot and the goal, \(d_{goal}^{t}\), and the distance between the robot and the patient, \(d_{p,r}^{t}\), are used as features in the reward function. The robot is rewarded for reducing \(d_{goal}^{t}\) and penalized for increasing \(d_{p,r}^{t}\) when it is close to humans or too far away from the patient. The robot is also punished for moving away from the patient beyond a certain threshold distance. The objective of the reward function is to lead the robot to move toward its target while ensuring the preservation of a secure separation from human entities and maintaining physical contact with the designated patient. If \(d_{p,r}^{t}\) is less than 0.25 m, it is considered that the robot and the patient are in contact. In accordance with the prescribed metric for the successful completion of the robot's objective, denoted as \(d_{goal}^{t}<0.25\) m, a reward of 10 is granted. This incentive scheme aligns with the assertion of [PERSON] et al. [53], i.e., reward structures with steep gradients are effective in shaping certain behaviors, both positively and negatively. To this end, a punitive penalty of \(-20\) is imposed as a negative reward for undesirable actions, while a positive reward of 10 is granted for accomplishing the desired objective. Notably, the reward for collision with objects is substantially more punitive, as collisions in the real world can lead to severe consequences. Consequently, the collision with objects is met with a steeply negative reward. The proposed model architecture encourages learning from the spatial and temporal features, and the reward function used in training by PPO enforces the rules of shared control and co-navigation. However, this model only focuses on dynamic obstacles. In the next subsection, this proposed model architecture is extended to include static obstacles as well.