# Transportation Mode Selection Using Reinforcement Learning in Simulation of Urban Mobility
## 4 Experimental Setup
### Baseline Models  
The baseline models used in this study were chosen to evaluate the effectiveness of the proposed Gaussian and Poisson reward mechanisms and to compare them with other methods. These models represent different types of approaches to travel mode choice and are used as reference points to more clearly demonstrate the performance advantages of the proposed mechanisms. The results of the baseline models were analyzed in comparison with the proposed methods in different traffic density scenarios. This analysis provides an important basis to demonstrate the accuracy and generalizability of the model. B-coefficient Logistic regression is linked to utilization theory, where the mode chosen is based on users' preferences for certain variables. In other words, utilization theory uses individuals' beliefs based on their preferences and is widely used by economists to explain user choice behavior based on their preference rankings. The equation below shows the logit model derived based on the sigmoid function of logistic regression and is widely used in modeling travel mode choice [42]. \[P(Y_{i})=\frac{1}{1+e^{-(B_{0}+B_{1}X_{1i}+B_{2}X_{2i}+\cdots+B_{n}X_{m})}} \tag{5}\]  
where  
* \(P(Y_{i})\) is the predicted probability that \(Y\) is true for case \(i\);
* \(e\) is a mathematical constant;
* \(B_{0}\) is a constant value estimated from the data;
* \(B_{1}\) is predicted from data B-coefficient;
* \(X_{i}\) is the observed score on variable \(X\) for case \(i\). The Random Utility Model [43], with two subgroupings (nests), determines a choice via the combination of the characteristics of the mode and the mode available. The utility function of the nested logit model has common unobserved characteristics (errors). Equation (6) shows the utility function for nested logit (NL) and cross-nested logit (CNL) [44]. \[U_{em}^{n}=V_{e}^{n}+V_{m}^{n}+V_{em}^{n}+\epsilon_{e}^{n}+\epsilon_{em}^{n} \tag{6}\]  
where  
Figure 4: Example traffic scenario in the SUMO simulation environment. Different colors represent various types of vehicles or traffic states: green indicates moving vehicles, yellow represents vehicles slowing down, and red highlights vehicles that are stopped or waiting. \(e\) is the existing mode;  
\(m\) is the travel mode;  
\(V^{n}_{e}\) is the deterministic component of the utility of the current mode;  
\(V^{m}_{m}\) is the deterministic component of the utility of the mode;  
\(V^{n}_{em}\) is the deterministic component of the co-benefit between the current mode and the mode;  
\(e^{n}_{e}\) is the error term for present or absent modes (Gumbel distribution);  
\(e^{n}_{em}\) is the error term for the combination of two subgroups (Gumbel distribution). The Reinforcement Learning Reward Model [45] is a reward system with time included in the reward used. Observed travel time, pre-event waiting time, and late arrival time are discretized and added to the reward as a penalty. The discretization is configurable and is set to 5 min in the experiments. The reward function for each agent is defined as follows:  
\[\text{Reward}= \begin{cases}1-ott&\text{if on time,}\\ 0-(ott+t_{\text{wait}})\cdot w_{\text{wait}}&\text{if too early,}\\ 0-(ott+t_{\text{late}})\cdot w_{\text{late}}&\text{if too late,}\\ 0-(ett_{\text{max}}\cdot 2)\cdot w_{\text{late}}&\text{if never arrived.}\end{cases}  \tag{7}\]  
If the agent shows up on time (that is, within 15 min prior to the event), the sole penalty is the observed travel time (\(ott\)), which measures how long it takes to get there. The waiting time is enhanced by the waiting time multiplier after being added to \(ott\) if the agent comes too early. \(w_{wait}\) = 1.0 is set for all experiments. The time from the event time to the destination is used to calculate the lateness (\(t_{late}\)) if the agent arrives too late. Similar to early arrival, \(t_{late}\) is increased by the lateness multiplier (\(w_{late}\)) after being added to \(ott\) if the agent arrives too late and chooses the worst kind of transportation at the event time.