# Transportation Mode Selection Using Reinforcement Learning in Simulation of Urban Mobility
## 4 Experimental Setup
### Environment and Ground Rules
The main purpose of this algorithm is to apply a reward/penalty system to find which transportation mode is best in terms of reaching the destination. In the initial state, s for all states and a for actions are set to zero. Certain states are observed, and an action is performed using the epsilon-greedy strategy. Here, the main criterion is the distance covered by the method to be used for transportation and, in addition, the time to reach the event area. Depending on the arrival time, a penalty is applied in cases of early or late arrival. Thanks to this process, learning is provided to see the most suitable transportation mode. Since transportation modes do not have equal initial distances, this randomness will be provided in a fair manner with the given reward mechanism. At the beginning of the algorithm, the Q-values for all states and actions are reset. This ensures that each state and action pair is initially evaluated equally. This step specifies that the algorithm will repeat until a certain stopping criterion is reached. After observing the current state, the agent chooses an action using an \(e\)-greedy strategy. If a randomly chosen probability value is less than \(e\), the agent chooses an action at random. If a randomly chosen probability value is greater than \(e\), the agent chooses the action with the highest Q-value given the current state. After performing the chosen action, the agent observes the new state and reward. A penalty is applied when the vehicle's arrival time deviates from the scheduled time, either arriving earlier or later. The penalty is calculated based on the arrival time; for example, the penalty for arriving 2 min early or late is multiplied by 2, while the penalty for arriving 5 min early or late is multiplied by 5. These penalties are added to the vehicle's arrival time. The calculated penalty is applied by subtracting the current reward. The Q-values are updated using the observed reward and the expected future reward. The update formula is obtained by adding the difference between the reward multiplied by the learning rate and the expected future reward to the current Q-value. Here, the learning rate (\(\alpha\)) is a value between 0 and 1, and the discount factor (\(\gamma\)) determines the importance of future rewards over current rewards. This is also a value between 0 and 1. The current state is updated as the new state, and the algorithm repeats these steps until a certain stopping criterion is reached. This criterion can usually be the completion of a certain number of episodes or when the change in Q-values is very small. In this way, the algorithm performs the learning process by taking into account the penalties according to the arrival time and updating the Q-values in order to make better decisions in the future. ```
1:Initialize Q-values: \(Q(s,a)=0\) for all states \(s\) and actions \(a\). 2:repeat
3: Observe the current state \(s\). 4: Select an action \(a\) using \(\epsilon\)-greedy strategy:
5:if random probability \(<\epsilon\)then
6: Choose a random action. 7:else
8: Choose the action with the highest Q-value: \(a=\arg\max_{s^{\prime}}Q(s,a^{\prime})\). 9:endif
10: Execute the action \(a\) and observe the new state \(s^{\prime}\) and reward \(r\). 11:if using penalty model then
12:if arrival time is early or late then
13: Apply penalty based on how early or late: \[\text{Penalty}= \begin{cases}2\times\text{arrival time}&\text{if 2 min early/late}\\ 5\times\text{arrival time}&\text{if 5 min early/late}\\ 10\times\text{arrival time}&\text{if 10 min early/late}\\ 15\times\text{arrival time}&\text{if 15 min early/late}\end{cases} \]
14: Add penalty to the reward: \(r\gets r\) - Penalty
15:endif
16:elseif using Gaussian model then
17: Apply Gaussian reward adjustment: \[\text{gaussianReward}= \begin{cases}\text{arrivetime}-(\text{arrivetime}\times \text{reward})&\text{if on time}\\ \text{arrivetime}-(\text{arrivetime}\times\text{reward})&\text{if too early}\\ \text{arrivetime}-(\text{arrivetime}\times\text{reward})&\text{if too late}\\ \end{cases}