# Transportation Mode Selection Using Reinforcement Learning in Simulation of Urban Mobility
## 3 Materials and Methods
### Reinforcement Learning
By employing RL, real-time and flexible solutions can be developed for transportation mode selection. This approach not only identifies the best solution under current conditions but also establishes a system that can quickly adapt to future changes and continuously improve over time. A snapshot of the agent's current state can be represented in either continuous-time or discrete-time formats, depending on the temporal characteristics of the problem and environment. Rewards act as the feedback mechanism provided by the environment in response to the agent's actions. These rewards can take on either positive or negative values, reflecting the desirability or undesirability of the agent's decisions and behavior. Lastly, the policy plays a pivotal role in guiding the agent's decision-making process. It outlines how an agent, situated in state \(s_{i}\) at time \(t\) selects an action from the available set of actions. The policy can be designed as either deterministic or stochastic, representing the strategy or approach that the agent follows in navigating its environment. Table 1 shows the key hyperparameters of the Reinforcement Learning (RL) model used in the study. The hyperparameters represent critical settings that affect the learning process of the model and were carefully selected to optimize the performance of the model:  
The learning Rate (\(\alpha\)) determines how much learning the model performs at each step. A small learning rate (0.001) allows the model to learn slowly and steadily, thus avoiding sudden fluctuations. The discount Factor (\(\gamma\)) determines the impact of future rewards on the current decision. A value of 0.95 provides a balance between short-term and long-term \begin{table}
\begin{tabular}{c c c} \hline \hline
**Parameter** & **Symbol** & **Value** \\ \hline Learning Rate & \(\alpha\) & 0.001 \\ Discount Factor & \(\gamma\) & 0.95 \\ Epsilon Initial Value & \(\epsilon_{0}\) & 1.0 \\ Epsilon Decay Rate & \(\epsilon_{\text{decay}}\) & 0.9 \\ Maximum Steps per Episode & \(N_{\text{max}}\) & 1000 \\ \hline \hline \end{tabular}
\end{table} 
Table 1: Hyperparameters used in the reinforcement learning model. rewards. The epsilon initial value (\(\epsilon_{0}\)) defines the initial probability of discovery. A high initial value (1.0) allows the model to make a random discovery in the initial stages. The epsilon decay rate (\(\epsilon_{decay}\)) controls how the probability of discovery decreases with each iteration. A value of 0.9 ensures that the epsilon decays gradually and the model tends towards more stable strategies. The maximum steps per episode (\(N_{max}\)) specifies the maximum number of steps that can be performed in each episode. A limit of 1000 steps provides a suitable framework for the model to perform enough learning within an episode. Initially, all Q-values are set to zero or some arbitrary value:  
\[Q(s,a)=0\quad\text{for all states $s$ and actions $a$} \tag{1}\]  
Agent Interaction and Action Selection  
At each time step, the following occurs:  
1. Observe the current state, \(s\). 2. Select an action, \(a\). Typically, the \(\epsilon\)-greedy strategy is used:  
* With probability \(\epsilon\), choose a random action;
* With probability \(1-\epsilon\), choose the action with the highest Q-value:  
\[a=\arg\max_{a^{\prime}}Q(s,a^{\prime}) \tag{2}\]  
Executing the Action and Receiving the Reward  
The agent performs action \(a\), observes the new state \(s^{\prime}\), and receives reward \(r\). Updating values  
\[Q(s,a)\gets Q(s,a)+\alpha\left[r+\gamma\max_{a^{\prime}}Q(s^{\prime},a^{ \prime})-Q(s,a)\right] \tag{3}\]  
where  
* \(a\) is the learning rate (a value between 0 and 1);
* \(\gamma\) is the discount factor (a value between 0 and 1 that determines the importance of future rewards). Updating the State  
Update the current state to the new state as follows:  
\[s\gets s^{\prime} \tag{4}\]  
Iteration  
Summary of Formulation  
1. \(Q(s,a)=0\). 2. Repeat for each episode: 1. \(s=s_{0}\); 2. Repeat for each step of the episode: 1.