## 4 Proposed Model
### Model Architecture  
As base model architecture, we use the TD3-based implementation discussed in our previous work in [6]. However, we make slight changes to the layer layout and activation functions. Moreover, since we are also considering the incoming edge information from other agents, this information needs to be embedded and combined with the node information. To achieve this, we design a local edge information embedder sub-graph. Since the number of agents in the scene that may interact with the ego agent is dynamic, this sub-graph must be capable of including varied size inputs. We achieve this by batching all incoming edge information \(E\) into a single batch with \(n\) dimensions. This batch is then passed through a sequence of embedding layers consisting of sequential, fully connected and continuously differentiable exponential linear unit (CELU) [38] activation  
Figure 2: Example of observation preparation into node and edge information in a scene with three agents. First, a state observation is sampled from the replay buffer. Node and edge information is formed for each agent in its frame of reference to create an individual sample from each agent’s perspective. Each created sample is then used in a neural network to update the learned policy. layers. This process returns \(n\)-dimensional embedding of all incoming edges. In order to obtain single-dimension embedding information for all edges, we perform maximum value aggregation over all embeddings. This aggregation returns the maximal value of each feature embedding in a single vector representation of all other agents in the scene. The edge information embedder sub-graph structure is visualized in Figure 3.  
The input to the neural network consists of ego agent features and a batch of other agent features, as described in Section 4.1 for every robot in the scene. The actor part of the neural network embeds the ego agent features with a fully connected linear layer followed by CELU activation function. The batch of other agent features is embedded using the sub-graph. The ego agent and other agent embeddings are concatenated, followed by a fully connected layer and a CELU activation layer. Finally, another fully connected layer follows with a hyperbolic tangent (Tanh) activation function that constrains the output to \([-1,1]\) range. The actor-network output is a two-dimensional vector representing the obtained linear and angular velocities for the ego agent. The actor-network structure is depicted in Figure 4, and the layers are described in Table 1.  
Figure 4: Actor–network structure. Inputs of ego agent and other agent information are depicted in gray. Edge embedder described in Figure 5 is stylized in light red. Fully connected layers are represented in yellow and CELU in dark purple, with Tanh activation in light purple. Concatenation, which combines the ego agent features with embedded edge features, is represented with the green circle. Here, the action calculated in the actor network is also used as an input to calculate state–action pair value. It is passed directly into the TFC layer, which is depicted in a light green color. Figure 3: Edge embedder layer structure for \(n\)-dimensional input of other agent information. The edge information is passed through a series of fully connected layers, all sharing the same weights. Fully connected layers are represented in yellow and CELU in purple. Max aggregation (visualized in red) is used to aggregate the encoded information to a single-dimension output. The chosen TD3 implementation uses two critics to mitigate the value overestimation often present in deep deterministic policy gradient (DDPG) neural networks [39]. Therefore, we implement two critics with the same layer structure. The critic networks have a similar architecture to the actor network. We embed the ego agent features and other agent features with their respective embedders. After concatenation, we obtain a single-dimension embedding of all the state information.