## 4 Proposed System for Patient-Robot Co-Navigation
### Shared Control in Crowded Environments Using Reinforcement Learning
#### 4.2.1 Shared Control Considering Only Dynamic Obstacles (Model V1)
\(V^{t}\) can be defined as the concatenation of the outputs of \(RNN_{c}\) and \(RNN_{p}\), which are \([s^{t}_{ci},s^{t}_{c_{2}},s^{t}_{c_{3}},\ldots,s^{t}_{c_{n}},s^{t}_{p}]\). Let the output of \(RNN_{r}\) be \(s^{t}_{r}\). The attention mechanism assigns relative weights to the inputs, which means that the attention mechanism determines the inputs that it should focus on. The first step is that the inputs \(V^{t}\) should be projected into an embedding space. This is computed by multiplying it with the matrix \(W_{Q}\), and the output of the multiplication is \(Q^{t}\). \(s^{t}_{r}\) is the output of \(RNN_{r}\), whose inputs are \((p_{r,x},p_{r,y},v_{r,x},v_{r,y})\), i.e., the current position and velocity of the robot. The task is to calculate the relative importance of each human and patient by assigning weights to each of them. For example, if a human moves very close to the robot, the human should obtain a higher weight than other pedestrians. Likewise, if the patient is moving away from the robot, then the patient needs to be given a higher weight so that the robot adjusts its path and is in the vicinity of the robot. Therefore, \(s^{t}_{r}\) is multiplied  
Figure 4: Proposed framework for the patient–robot co-navigation in environments with dynamic obstacles (Model V1). with matrix \(W_{K}\) to convert it to the embedded space, and the output of the multiplication is \(K^{t}\). The attention is calculated as shown in Equation (3), where \(alpha^{t}\) represents the relative weights of the human and patient data that the model should immediately give attention to, and att\({}^{t}\) is the re-weighted \(V^{t}\). \[ \begin{array}{l}Q^{t}=V^{t}W_{Q},\\ K^{t}=s_{t}^{t}W_{K},\\ a^{t}=\text{softmax}\Big{(}\frac{1}{\sqrt{n+1}}Q^{t}\big{(}K^{t}\big{)}^{\top} \Big{)}\\ \text{att}^{t}=\big{(}V^{t}\big{)}^{\top}a^{t}\end{array}  \tag{3}\]  
Both \(W_{K}\) and \(W_{Q}\) are learned while training the model. The output of the attention module is att\({}^{t}\). This weighted vector att\({}^{t}\) is concatenated with the robot node features \(x^{t}=[p_{r,x},p_{r,y},v_{r,x},v_{r,y},x_{r,x},g_{r,y},v_{r,max},\theta,\rho_ {robot}]\) and passed as input to the final node RNN \(RNN_{n}\). The output of this RNN is \(o^{t}\); which is defined in the Equation (4) shown below:  
\[o^{t}=RNN_{n}(o^{t-1},[\text{att}^{t},x^{t}]) \tag{4}\]  
Subsequently, the output \(o^{t}\) is passed through a fully connected layer, resulting in the determination of both the value function \(V(s^{t})\) and the policy distribution \(\pi(a^{t}|s^{t})\). The value \(V(s^{t})\) is the model's projected value function for how good a state \(s^{t}\) is for the robot to be in. \(\pi(a^{t}|s^{t})\) is the action that the robot should take given a state \(s^{t}\), which in our case is the velocity the robot should take, which is \(v_{r,x},v_{r,y}\). The total number of trainable parameters for this model is 613294. The crowd navigation and shared control can be formulated into a reinforcement learning problem in the following way: A robot interacting with the environment can be modeled as an episodic Markov decision process (MDP). Assuming the whole environment starts from a random initial state at time \(t=0\), at every discrete point in time, denoted as \(t\), the robot operates by selecting an action, \(a^{t}\), based on its policy function, \(\pi(a^{t}|s^{t})\). The policy maps the current state, \(s^{t}\), to a probability distribution over the set of possible actions, from which the robot selects its action, where \(s_{t}\) denotes the states of the whole environment (robot, humans, and patients). As a result of executing its chosen action, the robot receives a reward and transitions to the next state. The state transition and reward represent the consequences of the action taken in the current state and serve as the basis for the robot's decision-making process in the subsequent time step.