## 4 DRL Problem Formulation
### Neural Network Model  
Our proposed neural network architecture utilizes BiGRU layers to process variable-length input sequences. The BiGRU consists of two GRU layers that capture valuable temporal dependencies, enabling the model to understand the underlying relationships within the data better. The GRU's internal mechanisms, namely the update and reset gates, enable the model to selectively update its hidden state based on the current input and the previous hidden state. The update gate, \(z_{t}\), determines the extent to which the previous hidden state should be preserved and is calculated as:  
\[z_{t}=\sigma(W_{z}\cdot[h_{t-1},x_{t}]), \tag{24}\]  
where \(\sigma\) is the sigmoid activation function, \(W_{z}\) is the weight matrix, \(h_{t-1}\) is the hidden state in previous time, and \(x_{t}\) is the input state. The reset gate, \(r_{t}\), determines how much of the past information to forget and is calculated as given below:  
\[r_{t}=\sigma(W_{r}\cdot[h_{t-1},x_{t}]), \tag{25}\]  
where \(W_{r}\) denotes the weighted matrix associated with the reset gate. The BiGRU processes the input sequence in a forward and backward pass, resulting in two final hidden states, \(h_{f}\) and \(h_{b}\), respectively. At each step, \(B(t)\) and \(O(t)\) are fed into each BiGRU module, and the hidden states are then used to generate fixed-length representations, \(h_{B}\) and \(h_{O}\), for the robot and human agents, respectively.  
\[h_{B}=BiGRU([h_{b},h_{f}(t-1)],B(t)), \tag{26}\]  
\[h_{O}=BiGRU([h_{b},h_{f}(t-1)],O(t)). \tag{27}\]  
Later, the outputs \(h_{B}\) and \(h_{O}\) are concatenated as the integrated fixed-length observation, \(\Omega\). \(\Omega\) is then used as the input data for the first Multi-Layer Perceptron (MLP) with ReLU activation for the crowd features extraction, \(f_{e}=MLP(\Omega)\).  
Subsequently, it is fed into the policy actor, \(\pi_{a}\), which consists of an MLP with ReLU activation, \(\pi_{a}=MLP(f_{e})\). The weights assigned to human features, \(w_{f}\), are determined by applying a softmax function to the initial feature set, \(\Omega\).  
\[w_{f}=\sum_{i=1}^{n}\text{softmax}(\mathbf{\Omega}_{i}) \tag{28}\]  
This weighted feature vector is then concatenated with the action probabilities, \(\pi_{a}\), to form a comprehensive state representation.  
The resulting representation is subsequently fed into the policy critic, \(\pi_{c}\), which utilizes a Multi-Layer Perceptron (MLP) with ReLU activation to estimate the state value, \(v=\pi_{c}(w_{f},\pi_{a})\).  
Figure 6 shows the overall integration inside Robot \(B\) between the sensors and the RL module. The observable state, \(O(t)\), is provided by the RADAR sensor, which captures the position and velocity of moving obstacles. The full state, \(F(t)\), is obtained using the LiDAR sensor, which provides the robot's localization through the AMCL package. Additionally, the odometry computes the robot's velocity and the local goal is created by the path planning from \(A^{*}\) algorithm. The local goal, set approximately one meter from the robot's position and the closest point in the list of waypoints, is continuously evaluated until the robot reaches the final waypoint. Combining \(O(t)\) and \(F(t)\) create the input for the Bi-GRU, allowing memory retention and improved information handling for decision making for Navigation \(N\). The neural network then acts based on the joint state and provides the optimal action for the robot in the current state. This process continues iteratively until the robot reaches its goal.