# Reinforcement Learning Environment for Advanced Vehicular Ad Hoc Networks Communication Systems
## 3 Environment Model  
Instead of considering only the distance between nodes, this work proposes a routing method that considers the signal strength between nodes, taking into account the physical obstacles that degrade the communication. This interferes with the lifetime of the chosen path, until one or more nodes on the path leave the signal range, making the route invalid. Traditional ad hoc routing systems such as DSDV and DSR [19; 20] choose a route depending on the number of hops. Although the selected path contains a limited number of intermediate nodes between the source and the destination, the distance between each pair is usually high. If intermediate nodes with shorter distances between them are selected, the chance of path longevity increases. In the proposed task offloading structure, the agent manages the route table between vehicles. Vehicles in range signal coverage are considered connected and learn the vehicle states in real-time. The state contains the current position, speed, and id of each vehicle. The environment is formed as follows:  
The state-space \(S\), consists of the current state of the graph (in terms of a multidimensional output of a pro-based pairwise signal strength model) and a multifaceted observation; The action space \(A\), which consists of the next hop of the graph; Transitions between states, governed by the deterministic process-based model and the distance sequence; The reward \(r\) encourages the lifetime of the path. Observation space: In order to not violate [PERSON]'s property [13], that is, we do not know where the vehicles are moving, the alternative is to save some observations from the past and use them as a state. After receiving the vehicle states, the agent feeds the position matrix, keeping the last four previous positions and updating the current position. We propose to keep together four successive locations and observe each state; thus, this preprocessing stacks four rows, resulting in the final state space in the column array. That is, it is composed of a matrix that has the combination of all possible communication pairs (links) based on the signal strength between the vehicles. The matrix is exemplified in Equation (1). \[S=\left\{ \begin{bmatrix}RSSI_{t}(id_{0},id_{1})&...&RSSI_{t}(id_{0},id_{n})\\ RSSI_{t-1}(id_{0},id_{1})&...&RSSI_{t-1}(id_{0},id_{n})\\ RSSI_{t-2}(id_{0},id_{1})&...&RSSI_{t-2}(id_{0},id_{n})\\ RSSI_{t-3}(id_{0},id_{1})&...&RSSI_{t-3}(id_{0},id_{n})\\ &\vdots\\ \begin{bmatrix}RSSI_{t}(id_{0},id_{1})&...&RSSI_{t}(id_{0},id_{n})\\ D_{t-3}(0,1)&...&D_{t-3}((n-1),n)\end{bmatrix}\end{bmatrix} \right\} \tag{1}\]  
where \(RSSI\) means the Received Signal Strength Indicator between the hosts, the meaning is described in the Table 1. The \(t\) is the current timestamp, \(id\) is the identification number of vehicles in a given area, and the number of columns of the matrix is the combination of the pairs of hosts. In this work, we consider that less than \(-70\) dBm of \(RSSI\) signal, we do not consider that the vehicles are connected. Action space: In this case, the action space is discrete and determined by the number of cars in a certain place, such as downtown. It is made up of automobiles that can serve as origin, destination, or intermediate nodes along the path. Reward: There are three zones in which the reward function is specified. The package's destination is represented by the target. When the chosen \(S\) state may be part of the route, the viability zone is identified. When the specified state does not exist or cannot be a part of the path (due to the \(RSSI\) between the current node and the next-hop being weak), the reward function will get the reward value if it reaches the success zone, that means, the target. The lightest life route of dynamic graphs will be discovered using reinforcement learning methods. Lifetime, Reset, Observation, and Step are the algorithm's four primary functions. The Lifetime function determines how long the specified path will take to complete. When the route loops, the Reset function is executed.